{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *New Evidence of Discrimination Among Local Election Officials*\n",
    "\n",
    "** THIS CODE, AND THIS PAPER HAVE NOT BEEN PEER REVIEWED **\n",
    "\n",
    "**Authors**\n",
    "\n",
    "<!--\n",
    "D. Alex Hughes, University of California, Berkeley\n",
    "Micah Gell-Redman, University of Georgia\n",
    "Natarajan Krishnaswami, University of California, Berkeley\n",
    "Diana Rodenberger, University of California, Berkeley\n",
    "Guillermo Monge, University of California, Berkeley\n",
    "Charles Crabtree, University of Michigan\n",
    "--> \n",
    "\n",
    "**Analysis Date** \n",
    "August, 2018\n",
    "\n",
    "This file produces the analysis that accompanies the paper \"Who else gets to Vote?\". The analysis in this file contains more than is directly reported in the presented file. As such, it is possible that some of this analysis did not undergo peer review, and should be considered as such. \n",
    "\n",
    "<a id=\"toc\"></a>\n",
    "**Table of Contents** \n",
    "1. [Exploratory Data Analysis](#eda)\n",
    "  1. [Randomization](#randomization) \n",
    "  2. [Response Rate by Experimental Condition](#rbc)\n",
    "  3. [Differences in Baseline Responses, by Geography (Non-Experimental)](#stateDiffs) \n",
    "  4. [Most and Least Responsive States (Non-Experimental)](#mostLeast)\n",
    "2. [Experimental Results](#experimentalResults)\n",
    "  1. [Differences in Response Rates by Ethnic Condition(s)](#experimentalResults) \n",
    "  2. [Estimating Treatment Effects using OLS Models](#ols)\n",
    "  3. [Improving Model Performance Using Blocks](#blocks) \n",
    "  4. [Improving Model Performance Using Blocks and Covariates](#covariates)\n",
    "  5. [Precision Weighted Treatment Effect Estimates](#precisionWeighted)\n",
    "  6. [Plot of Precision Weighted Estimates](#precisionWeightedPlot)\n",
    "3. [Heterogenenous Treatment Effects](#hte)\n",
    "  1. [Racial and Ethnic District Characteristics](#demos)\n",
    "  2. [Political Characteristics](#politics)\n",
    "  3. [VRA Section 5 Coverage](#vra5)\n",
    "4. [Open Tracker](#hit)\n",
    "  1. \n",
    "4. [Interruption by Notification](#interruption)\n",
    "  1. [Rates of Response as function of Time Since Sending](#rateOfResponse)\n",
    "  2. [Time taken for Response](#timeToRespond)\n",
    "  3. [No Difference in Estimated Effect From Email](#noInterruptionDifference)\n",
    "5. [Open Rates](#openRates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Please cite as: \n",
      "\n",
      " Hlavac, Marek (2018). stargazer: Well-Formatted Regression and Summary Statistics Tables.\n",
      " R package version 5.2.2. https://CRAN.R-project.org/package=stargazer \n",
      "\n",
      "\n",
      "Attaching package: ‘zoo’\n",
      "\n",
      "The following objects are masked from ‘package:base’:\n",
      "\n",
      "    as.Date, as.Date.numeric\n",
      "\n",
      "\n",
      "Attaching package: ‘lmtest’\n",
      "\n",
      "The following object is masked from ‘package:lfe’:\n",
      "\n",
      "    waldtest\n",
      "\n",
      "\n",
      "Attaching package: ‘lubridate’\n",
      "\n",
      "The following objects are masked from ‘package:data.table’:\n",
      "\n",
      "    hour, isoweek, mday, minute, month, quarter, second, wday, week,\n",
      "    yday, year\n",
      "\n",
      "The following object is masked from ‘package:base’:\n",
      "\n",
      "    date\n",
      "\n"
     ]
    }
   ],
   "source": [
    "require(data.table, quietly = TRUE)\n",
    "require(sandwich,   quietly = TRUE)\n",
    "require(lfe,        quietly = TRUE)\n",
    "require(stargazer,  quietly = TRUE)\n",
    "require(lmtest,     quietly = TRUE)\n",
    "require(magrittr,   quietly = TRUE)\n",
    "require(IRdisplay,  quietly = TRUE)\n",
    "require(survival,   quietly = TRUE)\n",
    "require(lubridate,  quietly = TRUE)\n",
    "require(repr,       quietly = TRUE)\n",
    "require(interflex,  quietly = TRUE)\n",
    "require(gridExtra,  quietly = TRUE)\n",
    "\n",
    "## set digits to print to screen\n",
    "options(\n",
    "    digits=3, \n",
    "    repr.plot.width=4, \n",
    "    repr.plot.height=4,\n",
    "    jupyter.plot_mimetypes = c(\"text/plain\", \"image/png\" )\n",
    ")\n",
    "\n",
    "rm(list = ls())\n",
    "\n",
    "interrupt = TRUE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "code_show=true; \n",
       "function code_toggle() {\n",
       " if (code_show){\n",
       " $('div.input').hide();\n",
       " } else {\n",
       " $('div.input').show();\n",
       " }\n",
       " code_show = !code_show\n",
       "} \n",
       "$( document ).ready(code_toggle);\n",
       "</script>\n",
       "The raw code for this IPython notebook is by default hidden for easier reading.\n",
       "To toggle on/off the raw code, click <a href='javascript:code_toggle()'>here</a>."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_html(\"<script>\n",
    "code_show=true; \n",
    "function code_toggle() {\n",
    " if (code_show){\n",
    " $('div.input').hide();\n",
    " } else {\n",
    " $('div.input').show();\n",
    " }\n",
    " code_show = !code_show\n",
    "} \n",
    "$( document ).ready(code_toggle);\n",
    "</script>\n",
    "The raw code for this IPython notebook is by default hidden for easier reading.\n",
    "To toggle on/off the raw code, click <a href='javascript:code_toggle()'>here</a>.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define functions to be used to clean and prepare data for analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "load.data <- function(f=\"./data/replication_data.csv\") {\n",
    "    ## just a load function that wraps fread without warnings. \n",
    "    options(warn = -1)\n",
    "    d = read.csv(f, stringsAsFactors = FALSE)\n",
    "    d = data.table(d)\n",
    "    d[ , GotResponse := GotResponse == \"TRUE\"]\n",
    "    d[ , f.blockID := as.factor(blockID)]\n",
    "}\n",
    "\n",
    "rses <- function(mod) {\n",
    "    ## convienience function to pull robust standard errors\n",
    "    ## for reporting. \n",
    "    sqrt(diag(vcovHC(mod)))\n",
    "}\n",
    "\n",
    "felm_rses <- function(mod) { \n",
    "    ## convienience function to pull robust standard errors \n",
    "    ## from a felm object for reporting \n",
    "    mod$STATS$GotResponse$rse\n",
    "}\n",
    "\n",
    "\n",
    "gg.pooled.ate <- function(ate1, ate2, se1, se2) {\n",
    "    ## green and gerber pooled ate\n",
    "    weight1 = (1/se1^2) / (1/se1^2 + 1/se2^2)\n",
    "    weight2 = (1/se2^2) / (1/se1^2 + 1/se2^2)\n",
    "    ## calculate the ate \n",
    "    ate = (weight1 * ate1) + (weight2 * ate2)\n",
    "    return(ate)\n",
    "}\n",
    "\n",
    "gg.pooled.se <- function(se1, se2) {\n",
    "    ## green and gerber pooled standard error \n",
    "    se = sqrt(1 / ( (1/se1^2) + (1/se2^2) ))\n",
    "    return(se)\n",
    "}\n",
    "\n",
    "create.survival.data <- function(cd=censoring.date) {\n",
    "    ## creates a data.table for survival analysis, \n",
    "    ## specifying the censoring data. \n",
    "    ## \n",
    "    ## load data \n",
    "    d=load.data()\n",
    "    ## clean the date field for use as a POSIX class time \n",
    "    d[ , Date := gsub(\"T\", \" \", Date)]\n",
    "    d[ , Date := gsub(\"-07:00\", \" EST\", Date)]\n",
    "    ## create the date fields for non-blank `Date`\n",
    "    d[Date != \"\" , Date.num := as.numeric(as.POSIXct(Date))]\n",
    "    d[ , Date := NULL]\n",
    "    d[ , Date := Date.num]\n",
    "    d[ , Date.num := NULL]\n",
    "    ## create the end of observation time for results that \n",
    "    ## received a reply.\n",
    "    d[ReplyDate != \"\", SurvStop := as.numeric(as.POSIXct(ReplyDate))]\n",
    "    d[GotResponse == 0, SurvStop := as.numeric(as.POSIXct(cd))]\n",
    "    d[ , time := SurvStop - Date]\n",
    "    ## create an outcome that scores whether a row received a response\n",
    "    d[, survRespond := GotResponse]\n",
    "}\n",
    "\n",
    "sem <- function(x) { \n",
    "    sqrt(var(x) / length(x))\n",
    "\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"eda\"></a>\n",
    "# 1. Exploratory Data Analysis \n",
    "[toc](#toc)\n",
    "\n",
    "What percent of individuals -- overall -- responded to our mailer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "setwd('~/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = load.data(f='./data/replication_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Response Rate</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.578</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       " Response Rate\\\\\n",
       "\\hline\n",
       "\t 0.578\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Response Rate | \n",
       "|---|\n",
       "| 0.578 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Response Rate\n",
       "1 0.578        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d[ , .(\"Response Rate\" = mean(GotResponse, na.rm = T))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"randomization\"></a>\n",
    "## 1.A Randomization Check \n",
    "[toc](#toc)\n",
    "\n",
    "We side with Mutz and Pemantle (2011), Humphreys, Sanchez de la Sierra, and van der Windt (2012). In the context of this experiment, assignment to treatment conditions is conducted at random, systematically, with a reasonably amount of data, and as the result of a transparent code base. As such, we have no *ex ante* expectations that randomization should be subject to a chance failure, and rather take the point of Mutz and Pemantle that the realization of our randomization procedure produces an instance of a random process. \n",
    "\n",
    "Nevertheless, in the next cell we report the balance on covariates, by experimental condition. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>ethnic_cue</th><th scope=col>Mean Density</th><th scope=col>SE Density</th><th scope=col>Mean Income</th><th scope=col>SE Income</th><th scope=col>Mean Black</th><th scope=col>SE Black</th><th scope=col>Mean Latino</th><th scope=col>SE Latino</th><th scope=col>Mean Obama</th><th scope=col>SE Obama</th><th scope=col>Mean VRA</th><th scope=col>SE VRA</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0       </td><td>1.86    </td><td>0.0193  </td><td>0.0441  </td><td>0.000680</td><td>0.0434  </td><td>0.00257 </td><td>0.0551  </td><td>0.00254 </td><td>-0.0629 </td><td>0.00674 </td><td>0.120   </td><td>0.00811 </td></tr>\n",
       "\t<tr><td>1       </td><td>1.85    </td><td>0.0197  </td><td>0.0448  </td><td>0.000698</td><td>0.0433  </td><td>0.00267 </td><td>0.0545  </td><td>0.00253 </td><td>-0.0611 </td><td>0.00675 </td><td>0.117   </td><td>0.00803 </td></tr>\n",
       "\t<tr><td>2       </td><td>1.85    </td><td>0.0197  </td><td>0.0447  </td><td>0.000716</td><td>0.0442  </td><td>0.00268 </td><td>0.0557  </td><td>0.00265 </td><td>-0.0602 </td><td>0.00679 </td><td>0.120   </td><td>0.00810 </td></tr>\n",
       "\t<tr><td>3       </td><td>1.84    </td><td>0.0195  </td><td>0.0448  </td><td>0.000660</td><td>0.0434  </td><td>0.00258 </td><td>0.0541  </td><td>0.00255 </td><td>-0.0649 </td><td>0.00683 </td><td>0.118   </td><td>0.00806 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllllllllll}\n",
       " ethnic\\_cue & Mean Density & SE Density & Mean Income & SE Income & Mean Black & SE Black & Mean Latino & SE Latino & Mean Obama & SE Obama & Mean VRA & SE VRA\\\\\n",
       "\\hline\n",
       "\t 0        & 1.86     & 0.0193   & 0.0441   & 0.000680 & 0.0434   & 0.00257  & 0.0551   & 0.00254  & -0.0629  & 0.00674  & 0.120    & 0.00811 \\\\\n",
       "\t 1        & 1.85     & 0.0197   & 0.0448   & 0.000698 & 0.0433   & 0.00267  & 0.0545   & 0.00253  & -0.0611  & 0.00675  & 0.117    & 0.00803 \\\\\n",
       "\t 2        & 1.85     & 0.0197   & 0.0447   & 0.000716 & 0.0442   & 0.00268  & 0.0557   & 0.00265  & -0.0602  & 0.00679  & 0.120    & 0.00810 \\\\\n",
       "\t 3        & 1.84     & 0.0195   & 0.0448   & 0.000660 & 0.0434   & 0.00258  & 0.0541   & 0.00255  & -0.0649  & 0.00683  & 0.118    & 0.00806 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "ethnic_cue | Mean Density | SE Density | Mean Income | SE Income | Mean Black | SE Black | Mean Latino | SE Latino | Mean Obama | SE Obama | Mean VRA | SE VRA | \n",
       "|---|---|---|---|\n",
       "| 0        | 1.86     | 0.0193   | 0.0441   | 0.000680 | 0.0434   | 0.00257  | 0.0551   | 0.00254  | -0.0629  | 0.00674  | 0.120    | 0.00811  | \n",
       "| 1        | 1.85     | 0.0197   | 0.0448   | 0.000698 | 0.0433   | 0.00267  | 0.0545   | 0.00253  | -0.0611  | 0.00675  | 0.117    | 0.00803  | \n",
       "| 2        | 1.85     | 0.0197   | 0.0447   | 0.000716 | 0.0442   | 0.00268  | 0.0557   | 0.00265  | -0.0602  | 0.00679  | 0.120    | 0.00810  | \n",
       "| 3        | 1.84     | 0.0195   | 0.0448   | 0.000660 | 0.0434   | 0.00258  | 0.0541   | 0.00255  | -0.0649  | 0.00683  | 0.118    | 0.00806  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  ethnic_cue Mean Density SE Density Mean Income SE Income Mean Black SE Black\n",
       "1 0          1.86         0.0193     0.0441      0.000680  0.0434     0.00257 \n",
       "2 1          1.85         0.0197     0.0448      0.000698  0.0433     0.00267 \n",
       "3 2          1.85         0.0197     0.0447      0.000716  0.0442     0.00268 \n",
       "4 3          1.84         0.0195     0.0448      0.000660  0.0434     0.00258 \n",
       "  Mean Latino SE Latino Mean Obama SE Obama Mean VRA SE VRA \n",
       "1 0.0551      0.00254   -0.0629    0.00674  0.120    0.00811\n",
       "2 0.0545      0.00253   -0.0611    0.00675  0.117    0.00803\n",
       "3 0.0557      0.00265   -0.0602    0.00679  0.120    0.00810\n",
       "4 0.0541      0.00255   -0.0649    0.00683  0.118    0.00806"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d[ , .(\"Mean Density\" = mean(dens_log), \n",
    "       \"SE Density\"   = sem(dens_log), \n",
    "       \"Mean Income\"  = mean(pct_inc2pov_150),\n",
    "       \"SE Income\"    = sem(pct_inc2pov_150),\n",
    "       \"Mean Black\"   = mean(pct_race_black), \n",
    "       \"SE Black\"     = sem(pct_race_black), \n",
    "       \"Mean Latino\"  = mean(pct_latino), \n",
    "       \"SE Latino\"    = sem(pct_latino),\n",
    "       \"Mean Obama\"   = mean(obama_margin), \n",
    "       \"SE Obama\"     = sem(obama_margin), \n",
    "       \"Mean VRA\"     = mean(vra_county_state), \n",
    "       \"SE VRA\"       = sem(vra_county_state)), \n",
    "  keyby = .(ethnic_cue)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"rbc\"></a>\n",
    "## 1.B Response Rate by Experimental Condition \n",
    "[toc](#toc)\n",
    "\n",
    "The primary experimental test in this paper examines whether the response rate of registrars is different, conditional on the racial or ethnic cue that was sent to the registrars. To test this, a simple difference between the response rate (mean of responses per sent message) within experiental groups estimates the ATE. The following two tables report the mean response rate, standard error of that mean, and total number of subjecst in the condition. \n",
    "\n",
    "The first table reports a split between a white name and a racially or ethnically minority name. The second reports a split between white, Latino, Black, and Arab names. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "rr_by_minority <- d %>% \n",
    "  .[ , .(meanResponse = mean(GotResponse),\n",
    "         se = sqrt(mean(GotResponse) * (1-mean(GotResponse)) / .N), \n",
    "         n = .N), \n",
    "    keyby = .(ethnic_cue != 0)] %>% \n",
    "  .[ , .(\"Ethnic Cue\" = ethnic_cue, \n",
    "         \"Response Rate\" = meanResponse, \n",
    "         \"Standard Error\" = se, \n",
    "         \"N\" = n)] %>% \n",
    "  .[ , \"Ethnic Cue\" := c(\"White\", \"Minority\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Ethnic Cue Response Rate Standard Error    N\n",
      "1:      White         0.613        0.01213 1611\n",
      "2:   Minority         0.566        0.00713 4828\n"
     ]
    }
   ],
   "source": [
    "print(rr_by_minority)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "rr_by_condition <- d %>%\n",
    "  .[ , .(meanResponse = mean(GotResponse),\n",
    "         se = sqrt(mean(GotResponse) * (1-mean(GotResponse)) / .N), \n",
    "         n = .N),\n",
    "    keyby = .(ethnic_cue)] %>% \n",
    "  .[ , .(\"Ethnic Cue\" = ethnic_cue, \n",
    "         \"Response Rate\" = meanResponse, \n",
    "         \"Standard Error\" = se, \n",
    "         \"N\" = n)] %>% \n",
    "  .[ , \"Ethnic Cue\" := c(\"White\", \"Latino\", \"Black\", \"Arab\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Ethnic Cue Response Rate Standard Error    N\n",
      "1:      White         0.613         0.0121 1611\n",
      "2:     Latino         0.584         0.0123 1609\n",
      "3:      Black         0.614         0.0121 1613\n",
      "4:       Arab         0.501         0.0125 1606\n"
     ]
    }
   ],
   "source": [
    "print(rr_by_condition)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = capture.output(\n",
    "    stargazer(rr_by_minority, \n",
    "          type     = \"latex\", \n",
    "          out      = \"./figures-tables/table1Minority.tex\",\n",
    "          summary  = FALSE, \n",
    "          title    = \"Response Rate by Minority Condition\", \n",
    "          label    = \"tab:minorityRates\", \n",
    "          float    = FALSE, \n",
    "          table.placement = \"t\", \n",
    "          rownames = TRUE, \n",
    "          header   = FALSE, \n",
    "          flip     = TRUE)\n",
    "    )\n",
    "\n",
    "m = capture.output(\n",
    "    stargazer(rr_by_condition, \n",
    "          type     = \"latex\", \n",
    "          out      = \"./figures-tables/table1Minorities.tex\",\n",
    "          summary  = FALSE, \n",
    "          title    = \"Response Rate by Experimental Condition\", \n",
    "          label    = \"tab:rates\", \n",
    "          float    = FALSE,\n",
    "          table.placement = \"t\",\n",
    "          rownames = TRUE, \n",
    "          header   = FALSE, \n",
    "          flip     = TRUE)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"stateDiffs\"></a>\n",
    "## 1.C Were response rates meaningfully different in different parts of the country? \n",
    "[toc](#toc)\n",
    "\n",
    "\n",
    "While we are performing exploratory data analyis, we examine a number of *NON EXPERIMENTAL*: checks to see if things operate differently in different types of districts. \n",
    "\n",
    "Article I: Section 4 of the US Constitution provides that states shall determine the method of electing their representation. These determinations have led to considerable variation. But, has this variation had a meaningful impact on how the Registrars in different states respond to our stimulus? \n",
    "\n",
    "We first note that there is considerable variation in the magnitude of registrars per state; much more than other institutional features (e.g. Members of Congress)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>State</th><th scope=col>Number of Registrars</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Wisconsin    </td><td>1597         </td></tr>\n",
       "\t<tr><td>Michigan     </td><td>1387         </td></tr>\n",
       "\t<tr><td>Massachusetts</td><td> 351         </td></tr>\n",
       "\t<tr><td>Texas        </td><td> 252         </td></tr>\n",
       "\t<tr><td>Vermont      </td><td> 247         </td></tr>\n",
       "\t<tr><td>New Hampshire</td><td> 232         </td></tr>\n",
       "\t<tr><td>Virginia     </td><td> 133         </td></tr>\n",
       "\t<tr><td>Kentucky     </td><td> 120         </td></tr>\n",
       "\t<tr><td>Illinois     </td><td> 109         </td></tr>\n",
       "\t<tr><td>Kansas       </td><td> 105         </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " State & Number of Registrars\\\\\n",
       "\\hline\n",
       "\t Wisconsin     & 1597         \\\\\n",
       "\t Michigan      & 1387         \\\\\n",
       "\t Massachusetts &  351         \\\\\n",
       "\t Texas         &  252         \\\\\n",
       "\t Vermont       &  247         \\\\\n",
       "\t New Hampshire &  232         \\\\\n",
       "\t Virginia      &  133         \\\\\n",
       "\t Kentucky      &  120         \\\\\n",
       "\t Illinois      &  109         \\\\\n",
       "\t Kansas        &  105         \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "State | Number of Registrars | \n",
       "|---|---|---|---|---|---|---|---|---|---|\n",
       "| Wisconsin     | 1597          | \n",
       "| Michigan      | 1387          | \n",
       "| Massachusetts |  351          | \n",
       "| Texas         |  252          | \n",
       "| Vermont       |  247          | \n",
       "| New Hampshire |  232          | \n",
       "| Virginia      |  133          | \n",
       "| Kentucky      |  120          | \n",
       "| Illinois      |  109          | \n",
       "| Kansas        |  105          | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "   State         Number of Registrars\n",
       "1  Wisconsin     1597                \n",
       "2  Michigan      1387                \n",
       "3  Massachusetts  351                \n",
       "4  Texas          252                \n",
       "5  Vermont        247                \n",
       "6  New Hampshire  232                \n",
       "7  Virginia       133                \n",
       "8  Kentucky       120                \n",
       "9  Illinois       109                \n",
       "10 Kansas         105                "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d %>% \n",
    "  .[ , .(numberOfRegistrars = .N), keyby = .(state)] %>% \n",
    "  .[order(numberOfRegistrars, decreasing = TRUE)] %>%\n",
    "  .[1:10 , .(\"State\" = state, \"Number of Registrars\" = numberOfRegistrars)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the distribution of the response rate of these registrars? What is the distribution of responses on a state-by-state basis? In the next two histograms we report these statistics. The first histogram reports the response rate per state. The second reports the total number of responses per state (on a logged scale, due to the *Wisconsin* and *Michigan* having an outsized number of registrars). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "stateResponses <- d %>% \n",
    "  .[ , .(meanResponseRate = mean(GotResponse), \n",
    "         numRespond = sum(GotResponse)), \n",
    "    keyby = .(state)]\n",
    "\n",
    "pdf(\"./figures-tables/responseHistogram.pdf\", height = 5, width = 10)\n",
    "par(mfrow = c(1,2))\n",
    "source('./src/plot_function.R')\n",
    "hist(stateResponses[ , meanResponseRate], \n",
    "     col = \"black\", xlab = \"Response Rate\",  \n",
    "     main = \"Response Rate per State\")\n",
    "hist(stateResponses[ , log(numRespond)],   \n",
    "     col = \"black\", xlab = \"Log(Responses)\", \n",
    "     main = \"Responses per State\")\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../figures-tables/responseHistogram.pdf' width = 800>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"mostLeast\"></a>\n",
    "## 1.D What states were the most responsive? Which were the least responsive? \n",
    "[toc](#toc)\n",
    "\n",
    "In the following tables, we report the 8 most responsive states and the 8 least responsive states, as well as the total number of responses in each state. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>State</th><th scope=col>Response Rate</th><th scope=col>Number of Responses</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Delaware    </td><td>1.000       </td><td>  3         </td></tr>\n",
       "\t<tr><td>North Dakota</td><td>0.887       </td><td> 47         </td></tr>\n",
       "\t<tr><td>Florida     </td><td>0.851       </td><td> 57         </td></tr>\n",
       "\t<tr><td>Virginia    </td><td>0.812       </td><td>108         </td></tr>\n",
       "\t<tr><td>Minnesota   </td><td>0.805       </td><td> 70         </td></tr>\n",
       "\t<tr><td>Rhode Island</td><td>0.795       </td><td> 31         </td></tr>\n",
       "\t<tr><td>Wyoming     </td><td>0.783       </td><td> 18         </td></tr>\n",
       "\t<tr><td>Tennessee   </td><td>0.768       </td><td> 73         </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " State & Response Rate & Number of Responses\\\\\n",
       "\\hline\n",
       "\t Delaware     & 1.000        &   3         \\\\\n",
       "\t North Dakota & 0.887        &  47         \\\\\n",
       "\t Florida      & 0.851        &  57         \\\\\n",
       "\t Virginia     & 0.812        & 108         \\\\\n",
       "\t Minnesota    & 0.805        &  70         \\\\\n",
       "\t Rhode Island & 0.795        &  31         \\\\\n",
       "\t Wyoming      & 0.783        &  18         \\\\\n",
       "\t Tennessee    & 0.768        &  73         \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "State | Response Rate | Number of Responses | \n",
       "|---|---|---|---|---|---|---|---|\n",
       "| Delaware     | 1.000        |   3          | \n",
       "| North Dakota | 0.887        |  47          | \n",
       "| Florida      | 0.851        |  57          | \n",
       "| Virginia     | 0.812        | 108          | \n",
       "| Minnesota    | 0.805        |  70          | \n",
       "| Rhode Island | 0.795        |  31          | \n",
       "| Wyoming      | 0.783        |  18          | \n",
       "| Tennessee    | 0.768        |  73          | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  State        Response Rate Number of Responses\n",
       "1 Delaware     1.000           3                \n",
       "2 North Dakota 0.887          47                \n",
       "3 Florida      0.851          57                \n",
       "4 Virginia     0.812         108                \n",
       "5 Minnesota    0.805          70                \n",
       "6 Rhode Island 0.795          31                \n",
       "7 Wyoming      0.783          18                \n",
       "8 Tennessee    0.768          73                "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "stateResponses %>% \n",
    "  .[order(meanResponseRate, decreasing = TRUE), ] %>% \n",
    "  .[1:8, ] %>% \n",
    "  .[ , .(\"State\" = state, \n",
    "         \"Response Rate\" = meanResponseRate, \n",
    "         \"Number of Responses\" = numRespond)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What states were the *least* responsive? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>State</th><th scope=col>Response Rate</th><th scope=col>Number of Responses</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Vermont      </td><td>0.389        </td><td> 96          </td></tr>\n",
       "\t<tr><td>Mississippi  </td><td>0.438        </td><td> 35          </td></tr>\n",
       "\t<tr><td>Utah         </td><td>0.448        </td><td> 13          </td></tr>\n",
       "\t<tr><td>Michigan     </td><td>0.456        </td><td>632          </td></tr>\n",
       "\t<tr><td>Arkansas     </td><td>0.467        </td><td> 35          </td></tr>\n",
       "\t<tr><td>Connecticut  </td><td>0.495        </td><td> 48          </td></tr>\n",
       "\t<tr><td>California   </td><td>0.509        </td><td> 28          </td></tr>\n",
       "\t<tr><td>Nevada       </td><td>0.529        </td><td>  9          </td></tr>\n",
       "\t<tr><td>Oklahoma     </td><td>0.532        </td><td> 41          </td></tr>\n",
       "\t<tr><td>Iowa         </td><td>0.545        </td><td> 54          </td></tr>\n",
       "\t<tr><td>New Hampshire</td><td>0.552        </td><td>128          </td></tr>\n",
       "\t<tr><td>Wisconsin    </td><td>0.562        </td><td>897          </td></tr>\n",
       "\t<tr><td>South Dakota </td><td>0.578        </td><td> 37          </td></tr>\n",
       "\t<tr><td>Oregon       </td><td>0.600        </td><td> 21          </td></tr>\n",
       "\t<tr><td>Montana      </td><td>0.607        </td><td> 34          </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " State & Response Rate & Number of Responses\\\\\n",
       "\\hline\n",
       "\t Vermont       & 0.389         &  96          \\\\\n",
       "\t Mississippi   & 0.438         &  35          \\\\\n",
       "\t Utah          & 0.448         &  13          \\\\\n",
       "\t Michigan      & 0.456         & 632          \\\\\n",
       "\t Arkansas      & 0.467         &  35          \\\\\n",
       "\t Connecticut   & 0.495         &  48          \\\\\n",
       "\t California    & 0.509         &  28          \\\\\n",
       "\t Nevada        & 0.529         &   9          \\\\\n",
       "\t Oklahoma      & 0.532         &  41          \\\\\n",
       "\t Iowa          & 0.545         &  54          \\\\\n",
       "\t New Hampshire & 0.552         & 128          \\\\\n",
       "\t Wisconsin     & 0.562         & 897          \\\\\n",
       "\t South Dakota  & 0.578         &  37          \\\\\n",
       "\t Oregon        & 0.600         &  21          \\\\\n",
       "\t Montana       & 0.607         &  34          \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "State | Response Rate | Number of Responses | \n",
       "|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n",
       "| Vermont       | 0.389         |  96           | \n",
       "| Mississippi   | 0.438         |  35           | \n",
       "| Utah          | 0.448         |  13           | \n",
       "| Michigan      | 0.456         | 632           | \n",
       "| Arkansas      | 0.467         |  35           | \n",
       "| Connecticut   | 0.495         |  48           | \n",
       "| California    | 0.509         |  28           | \n",
       "| Nevada        | 0.529         |   9           | \n",
       "| Oklahoma      | 0.532         |  41           | \n",
       "| Iowa          | 0.545         |  54           | \n",
       "| New Hampshire | 0.552         | 128           | \n",
       "| Wisconsin     | 0.562         | 897           | \n",
       "| South Dakota  | 0.578         |  37           | \n",
       "| Oregon        | 0.600         |  21           | \n",
       "| Montana       | 0.607         |  34           | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "   State         Response Rate Number of Responses\n",
       "1  Vermont       0.389          96                \n",
       "2  Mississippi   0.438          35                \n",
       "3  Utah          0.448          13                \n",
       "4  Michigan      0.456         632                \n",
       "5  Arkansas      0.467          35                \n",
       "6  Connecticut   0.495          48                \n",
       "7  California    0.509          28                \n",
       "8  Nevada        0.529           9                \n",
       "9  Oklahoma      0.532          41                \n",
       "10 Iowa          0.545          54                \n",
       "11 New Hampshire 0.552         128                \n",
       "12 Wisconsin     0.562         897                \n",
       "13 South Dakota  0.578          37                \n",
       "14 Oregon        0.600          21                \n",
       "15 Montana       0.607          34                "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "stateResponses %>% \n",
    "  .[order(meanResponseRate, decreasing = FALSE), ] %>%\n",
    "  .[1:15, ] %>% \n",
    "  .[ , .(\"State\" = state, \n",
    "         \"Response Rate\" = meanResponseRate, \n",
    "         \"Number of Responses\" = numRespond)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is considerable difference in the responsiveness of states. In a number of states nearly 90% of the emails sent are replied to; in other states, fewer than 50% of emails are responded to. \n",
    "\n",
    "One of the primary heterogeneities that we have proposed to examine in this work is whether Registrars in states previously covered by VRA Section 5 protections react differently to experimental stimulus than Registrars in states previously not covered by VRA Section 5 protections. At this point, and throughout, it is important to reiterate that this form of comparison is non-experimental. Any differences in observed responsiveness across a non-experimentally assigning split *may* or *may not* be causally associated with the splitting factor. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"baselineDiff\"></a>\n",
    "## Baseline Differences in Responsiveness \n",
    "To set the expectation for later heterogeneous treatment effects that we will examine, here we report the **non experimental** differences in the response rate per state. To do so, we estimate differences in the response rate based on:\n",
    "\n",
    "1. Past VRA status\n",
    "2. Current Latino population\n",
    "3. Current Black population\n",
    "\n",
    "We note that it is not possible to make a similar split based on Arab residents. This information is not currently collected by the US Census Bureau. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we report this split based on whether a registrar worked in a district that had previously been covered by Section 5 of the VRA. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Previously Covered by VRA</th><th scope=col>Response Rate</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>No   </td><td>0.567</td></tr>\n",
       "\t<tr><td>Yes  </td><td>0.658</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Previously Covered by VRA & Response Rate\\\\\n",
       "\\hline\n",
       "\t No    & 0.567\\\\\n",
       "\t Yes   & 0.658\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Previously Covered by VRA | Response Rate | \n",
       "|---|---|\n",
       "| No    | 0.567 | \n",
       "| Yes   | 0.658 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Previously Covered by VRA Response Rate\n",
       "1 No                        0.567        \n",
       "2 Yes                       0.658        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d %>% \n",
    "  .[ , .(responseRate = mean(GotResponse)), \n",
    "    keyby = .(\"Previously Covered by VRA\" = vra_county_state)] %>%\n",
    "  .[ , .(\"Previously Covered by VRA\" = c(\"No\", \"Yes\"), \n",
    "         \"Response Rate\" = responseRate) ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, we report this split based on whether a registrars worked in a district with a Latino population larger than the median Latino population for all the districts in our sample. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>&gt; Median Latino Population</th><th scope=col>Response Rate</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>FALSE</td><td>0.533</td></tr>\n",
       "\t<tr><td> TRUE</td><td>0.623</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " > Median Latino Population & Response Rate\\\\\n",
       "\\hline\n",
       "\t FALSE & 0.533\\\\\n",
       "\t  TRUE & 0.623\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "> Median Latino Population | Response Rate | \n",
       "|---|---|\n",
       "| FALSE | 0.533 | \n",
       "|  TRUE | 0.623 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  > Median Latino Population Response Rate\n",
       "1 FALSE                      0.533        \n",
       "2  TRUE                      0.623        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d %>% \n",
    "  .[ , .(responseRate = mean(GotResponse)), \n",
    "    keyby = .(pct_latino > median(pct_latino, na.rm = TRUE))] %>% \n",
    "  .[ , .(\"> Median Latino Population\" = pct_latino, \n",
    "         \"Response Rate\" = responseRate)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Third, we report this split based on whether a registrar worked in a district with a Black population larger than the median Black population for all the districts in our sample. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>&gt; Median Black Population</th><th scope=col>Response Rate</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>FALSE</td><td>0.512</td></tr>\n",
       "\t<tr><td> TRUE</td><td>0.644</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " > Median Black Population & Response Rate\\\\\n",
       "\\hline\n",
       "\t FALSE & 0.512\\\\\n",
       "\t  TRUE & 0.644\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "> Median Black Population | Response Rate | \n",
       "|---|---|\n",
       "| FALSE | 0.512 | \n",
       "|  TRUE | 0.644 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  > Median Black Population Response Rate\n",
       "1 FALSE                     0.512        \n",
       "2  TRUE                     0.644        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d %>% \n",
    "  .[ , .(responseRate = mean(GotResponse)), \n",
    "    keyby = .(pct_race_black > median(pct_race_black, na.rm = TRUE))] %>% \n",
    "  .[ , .(\"> Median Black Population\" = pct_race_black, \n",
    "         \"Response Rate\" = responseRate)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"experimentalResults\"></a>\n",
    "# 2. Experimental Results \n",
    "## 2.A Differences in Response Rates by Ethnic Condition(s) \n",
    "[toc](#toc)\n",
    "\n",
    "\n",
    "\n",
    "Here, we report the primary findings of this paper. \n",
    "\n",
    "First, when a question about what documents are necessary to provide in order to vote is asked by a minority identity (*not* a white identity) Registrars are less likely to respond to that query. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Ethnic Cue</th><th scope=col>Response Rate</th><th scope=col>Standard Error</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>White   </td><td>0.613   </td><td>0.01214 </td></tr>\n",
       "\t<tr><td>Minority</td><td>0.566   </td><td>0.00713 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " Ethnic Cue & Response Rate & Standard Error\\\\\n",
       "\\hline\n",
       "\t White    & 0.613    & 0.01214 \\\\\n",
       "\t Minority & 0.566    & 0.00713 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Ethnic Cue | Response Rate | Standard Error | \n",
       "|---|---|\n",
       "| White    | 0.613    | 0.01214  | \n",
       "| Minority | 0.566    | 0.00713  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Ethnic Cue Response Rate Standard Error\n",
       "1 White      0.613         0.01214       \n",
       "2 Minority   0.566         0.00713       "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b> Causal effect of contact by a minority sender: </b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "-4.7004"
      ],
      "text/latex": [
       "-4.7004"
      ],
      "text/markdown": [
       "-4.7004"
      ],
      "text/plain": [
       "[1] -4.7"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d %>% \n",
    "  .[ , .(meanResponse = mean(GotResponse), \n",
    "         se = sqrt(var(GotResponse) / .N)), \n",
    "    keyby = .(\"Ethnic Cue\" = ethnic_cue != 0)] %>%\n",
    "  .[ , .(\"Ethnic Cue\" = c(\"White\", \"Minority\"), \n",
    "         \"Response Rate\" = meanResponse, \n",
    "         \"Standard Error\" = se)]\n",
    "\n",
    "display_html(\"<b> Causal effect of contact by a minority sender: </b>\")\n",
    "d %>% \n",
    "  .[ , .(meanResponse = mean(GotResponse)), \n",
    "    keyby = .(I(ethnic_cue!=0))] %>%\n",
    "  .[ , round(diff(meanResponse) * 100, 4)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, these differences depend on the ethnic condition that is provided to the Registrar. There is no evidence to suggest that black and white identities are responded to at different rates. Indeed, black and white mailers are responded to at nearly identical rates. However, Latino identities are responded to at about 3% lower rates than white identities; and, Arab identities are responded to at about 10% lower rates than white identities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Ethnic Cue</th><th scope=col>Mean Response</th><th scope=col>Standard Error</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>White </td><td>0.613 </td><td>0.0121</td></tr>\n",
       "\t<tr><td>Latino</td><td>0.584 </td><td>0.0123</td></tr>\n",
       "\t<tr><td>Black </td><td>0.614 </td><td>0.0121</td></tr>\n",
       "\t<tr><td>Arab  </td><td>0.501 </td><td>0.0125</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " Ethnic Cue & Mean Response & Standard Error\\\\\n",
       "\\hline\n",
       "\t White  & 0.613  & 0.0121\\\\\n",
       "\t Latino & 0.584  & 0.0123\\\\\n",
       "\t Black  & 0.614  & 0.0121\\\\\n",
       "\t Arab   & 0.501  & 0.0125\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Ethnic Cue | Mean Response | Standard Error | \n",
       "|---|---|---|---|\n",
       "| White  | 0.613  | 0.0121 | \n",
       "| Latino | 0.584  | 0.0123 | \n",
       "| Black  | 0.614  | 0.0121 | \n",
       "| Arab   | 0.501  | 0.0125 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Ethnic Cue Mean Response Standard Error\n",
       "1 White      0.613         0.0121        \n",
       "2 Latino     0.584         0.0123        \n",
       "3 Black      0.614         0.0121        \n",
       "4 Arab       0.501         0.0125        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d %>% \n",
    "  .[ , .(meanResponse = mean(GotResponse), \n",
    "         se = sqrt(var(GotResponse) / .N)), \n",
    "    keyby = .(\"Ethnic Cue\" = ethnic_cue)] %>% \n",
    "  .[ , .(\"Ethnic Cue\" = c(\"White\", \"Latino\", \"Black\", \"Arab\"), \n",
    "         \"Mean Response\" = meanResponse, \n",
    "         \"Standard Error\" = se)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nonparametric Wilcoxon Rank-Sum Tests for the difference in location of ranked means, as well as t-tests for differences in means provide evidence in support of lower rates of response to Latino and Arab names (vis-a-vis white names), but do not provide such support for Black names.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Test</th><th scope=col>P.Value</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>All Minorities v. White</td><td>0.001                  </td></tr>\n",
       "\t<tr><td>Latino v. White        </td><td>0.086                  </td></tr>\n",
       "\t<tr><td>Black v. White         </td><td>0.949                  </td></tr>\n",
       "\t<tr><td>Arab v. White          </td><td>0.000                  </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Test & P.Value\\\\\n",
       "\\hline\n",
       "\t All Minorities v. White & 0.001                  \\\\\n",
       "\t Latino v. White         & 0.086                  \\\\\n",
       "\t Black v. White          & 0.949                  \\\\\n",
       "\t Arab v. White           & 0.000                  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Test | P.Value | \n",
       "|---|---|---|---|\n",
       "| All Minorities v. White | 0.001                   | \n",
       "| Latino v. White         | 0.086                   | \n",
       "| Black v. White          | 0.949                   | \n",
       "| Arab v. White           | 0.000                   | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Test                    P.Value\n",
       "1 All Minorities v. White 0.001  \n",
       "2 Latino v. White         0.086  \n",
       "3 Black v. White          0.949  \n",
       "4 Arab v. White           0.000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wilcox_minority = d[ , wilcox.test(GotResponse*1 ~ I(ethnic_cue != 0))]\n",
    "wilcox_latino   = d[ethnic_cue %in% c(0,1), wilcox.test(GotResponse*1 ~ I(ethnic_cue == 1))]\n",
    "wilcox_black    = d[ethnic_cue %in% c(0,2), wilcox.test(GotResponse*1 ~ I(ethnic_cue == 2))]\n",
    "wilcox_arab     = d[ethnic_cue %in% c(0,3), wilcox.test(GotResponse*1 ~ I(ethnic_cue == 3))]\n",
    "\n",
    "comparison_table = data.frame( \n",
    "  Test = c(\"All Minorities v. White\", \"Latino v. White\",\n",
    "           \"Black v. White\", \"Arab v. White\"), \n",
    "  P.Value = round(\n",
    "      c(wilcox_minority$p.value, wilcox_latino$p.value, \n",
    "              wilcox_black$p.value, wilcox_arab$p.value), 3)\n",
    "    )\n",
    "comparison_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ols\"></a>\n",
    "## 2.B Estimating Treatment Effects using OLS Models \n",
    "[toc](#toc)\n",
    "\n",
    "OLS estimators -- even when estimated on dichotomous data -- are BLUE estimators of causal effects and are familiar to political scientists (Wooldridge, Angrist & Pischke). \n",
    "\n",
    "In this sub-section, we estimate models that provide the same information as provided in the difference-in-means estimators of the previous sub-section. \n",
    "\n",
    "For each model, we estimate regression coefficients using OLS and then estimate Heteroskedstic Consistent Standard Errors. Rather than assuming normality of the residuals of these models (an assumption certainly violated with dichotomous outcome data) we utilize the White robust SE estimate. In the analysis reported in the supplemental information, we note that changing the link function does not meaningfully change the point estimate or interpretation of model estimates. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "null_mod       <- d[ , lm(GotResponse ~ 1)] \n",
    "minority_mod   <- d[ , lm(GotResponse ~ 1 + I(ethnic_cue != 0))]\n",
    "minorities_mod <- d[ , lm(GotResponse ~ 1 + factor(ethnic_cue))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>F.Test</th><th scope=col>P.Value</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Non-White v. White       </td><td>0.001                    </td></tr>\n",
       "\t<tr><td>Minority Classes v. White</td><td>0.000                    </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " F.Test & P.Value\\\\\n",
       "\\hline\n",
       "\t Non-White v. White        & 0.001                    \\\\\n",
       "\t Minority Classes v. White & 0.000                    \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "F.Test | P.Value | \n",
       "|---|---|\n",
       "| Non-White v. White        | 0.001                     | \n",
       "| Minority Classes v. White | 0.000                     | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  F.Test                    P.Value\n",
       "1 Non-White v. White        0.001  \n",
       "2 Minority Classes v. White 0.000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "anova_minority   <- anova(null_mod, minority_mod, test = \"F\")\n",
    "anova_minorities <- anova(null_mod, minorities_mod, test = \"F\")\n",
    "\n",
    "comparison_table <- data.frame( \n",
    "  F.Test = c(\"Non-White v. White\", \"Minority Classes v. White\"), \n",
    "  P.Value = round(\n",
    "    c(anova_minority[2,6], anova_minorities[2,6]), 3)\n",
    ")\n",
    "comparison_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 1: OLS Treatment Effects \n",
    "We first estimate treatment effects using the OLS estimator and heteroskedastic-consistent (robust) standard errors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================\n",
      "                 Dependent variable:     \n",
      "             ----------------------------\n",
      "                     GotResponse         \n",
      "                  (1)            (2)     \n",
      "-----------------------------------------\n",
      "Minority       -0.047***                 \n",
      "                (0.014)                  \n",
      "                                         \n",
      "Latino                         -0.030*   \n",
      "                               (0.017)   \n",
      "                                         \n",
      "Black                           0.001    \n",
      "                               (0.017)   \n",
      "                                         \n",
      "Arab                          -0.113***  \n",
      "                               (0.017)   \n",
      "                                         \n",
      "Constant        0.613***      0.613***   \n",
      "                (0.012)        (0.012)   \n",
      "                                         \n",
      "-----------------------------------------\n",
      "Observations     6,439          6,439    \n",
      "R2               0.002          0.009    \n",
      "Adjusted R2      0.002          0.008    \n",
      "=========================================\n",
      "Note:         *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\n",
       "t test of coefficients:\n",
       "\n",
       "                    Estimate Std. Error t value Pr(>|t|)    \n",
       "(Intercept)           0.6133     0.0121   50.51  < 2e-16 ***\n",
       "factor(ethnic_cue)1  -0.0297     0.0173   -1.72    0.086 .  \n",
       "factor(ethnic_cue)2   0.0011     0.0172    0.06    0.949    \n",
       "factor(ethnic_cue)3  -0.1127     0.0174   -6.47  1.1e-10 ***\n",
       "---\n",
       "Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "stargazer(minority_mod, minorities_mod, \n",
    "          type = 'text',\n",
    "          covariate.labels = c(\"Minority\", \"Latino\", \"Black\", \"Arab\"), \n",
    "          se = list(rses(minority_mod), rses(minorities_mod)), \n",
    "          omit.stat = c(\"F\", \"ser\")\n",
    "         )\n",
    "coeftest(minorities_mod, vcovHC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"blocks\"></a>\n",
    "## 2.C Improve Model Performance Using Blocks \n",
    "[toc](#toc)\n",
    "\n",
    "To improve model performance, we estimate a within estimator using the blocks created before treatment was assigned. Because there are a large number of blocks, we utilize the `felm` call from the package `lfe` which uses a method of projection to remove the mean response rates within each block. Estimates from this method are identical to estimates that simply include block fixed effects. The treatment effect estimates using blocks are highly similar to the treatment effect estimates without blocks -- which is to be expected because treatment assignment is intentionally orthagonal to block -- though model efficiency is higher due to the decreases residual varaince in the model. \n",
    "\n",
    "Unblocked models are estimated as OLS regressions of the following equations: \n",
    "\n",
    "\\begin{align} \n",
    "Y_{i} & = \\beta_{0} + \\beta_{1} Minority + \\epsilon_{i} \\\\ \n",
    "Y_{i} & = \\beta_{0} + \\beta_{1} Latino + \\beta_{2} Black + \\beta_{3} Arab + \\epsilon_{i} \\\\ \n",
    "\\end{align}\n",
    "\n",
    "Blocked models are estimated as FELM regressions of the following *within-estimator* equations: \n",
    "\n",
    "\\begin{align} \n",
    "Y_{i} &= \\beta_{1} Minority + \\phi BlockID + \\epsilon_{i} \\\\\n",
    "Y_{i} &= \\beta_{1} Latino + \\beta_{2} Black + \\beta_{3} Arab + \\phi BlockID + \\epsilon_{i}\n",
    "\\end{align} \n",
    "\n",
    "Notably, with block-level fixed effects there is no sense in estimating an intercept, as the intercept is simply the omitted block. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "====================================================\n",
      "                       Dependent variable:          \n",
      "             ---------------------------------------\n",
      "                           GotResponse              \n",
      "                     OLS                felm        \n",
      "                (1)       (2)       (3)       (4)   \n",
      "----------------------------------------------------\n",
      "Minority     -0.047***           -0.047***          \n",
      "              (0.014)             (0.013)           \n",
      "                                                    \n",
      "Latino                  -0.030*             -0.030* \n",
      "                        (0.017)             (0.016) \n",
      "                                                    \n",
      "Black                    0.001               0.002  \n",
      "                        (0.017)             (0.017) \n",
      "                                                    \n",
      "Arab                   -0.113***           -0.113***\n",
      "                        (0.017)             (0.016) \n",
      "                                                    \n",
      "Constant     0.613***  0.613***                     \n",
      "              (0.012)   (0.012)                     \n",
      "                                                    \n",
      "----------------------------------------------------\n",
      "Block FE        No        No        Yes       Yes   \n",
      "Observations   6,439     6,439     6,439     6,439  \n",
      "R2             0.002     0.009     0.330     0.337  \n",
      "Adjusted R2    0.002     0.008     0.104     0.113  \n",
      "====================================================\n",
      "Note:                    *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "minority_mod_felm   <- d[ , felm(GotResponse ~ 1 + I(ethnic_cue != 0) | factor(blockID))]\n",
    "minorities_mod_felm <- d[ , felm(GotResponse ~ 1 + factor(ethnic_cue) | factor(blockID))]\n",
    "\n",
    "stargazer(minority_mod, minorities_mod, minority_mod_felm, minorities_mod_felm, \n",
    "          type = \"text\",  \n",
    "          se = list(rses(minority_mod), \n",
    "                    rses(minorities_mod), \n",
    "                    felm_rses(minority_mod_felm), \n",
    "                    felm_rses(minorities_mod_felm)),\n",
    "          covariate.labels = c(\"Minority\", \"Latino\", \"Black\", \"Arab\"), \n",
    "          add.lines = list(c(\"Block FE\", \"No\", \"No\", \"Yes\", \"Yes\")), \n",
    "          omit.stat = c(\"F\", \"ser\")\n",
    "          )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Briefly, we interpret these model coefficients in the following way: \n",
    "\n",
    "- Estimates of the causal effect of sending contact to registrars from white or minority sender names is unaffected by the decicision to use blocking in the estimating equation. \n",
    "- As expected, estimating these causal effects with blocks leads to small improvmeents in model efficiency.\n",
    "- Compared to being contacted by a white voter, being contacted by a minority voter causes a decrease in the response rate of 4.7 percent $(P < 0.0005)$. \n",
    "- Compared to being contacted by a white voter, being contacted by a Latino voter causes a decrease in the response rate of 3.0 percent $(P = 0.069)$.\n",
    "- Compared to being contacted by a white voter, being contacted by a Black voter does not change the response rate. \n",
    "- Compared to being contacted by a white voter, being contacted by an Arab voter causes a decrease in the response rate of 11.3 percent $(P < 6*10^-12)$. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"covariates\"></a>\n",
    "## 2.D Increased Precision with Blocks and Covariates \n",
    "[toc](#toc)\n",
    "\n",
    "One possobility is that we can further improve our estimates by using not only the block indicators but also the underlying data features that were used to create the blocks. In practice, however, including these features does not seem to have a demonstrable effect on the estimates. \n",
    "\n",
    "In these models, we estimate the following equations: \n",
    "\n",
    "\\begin{align} \n",
    "Y_{i} = \\beta_{0} & + \\beta_{1} Minority + \\gamma_{1}Population\\_Density + \\gamma_{2}SES\\_Indicator + \\gamma_{3} Pct\\_Black + \\gamma_{4}Pct\\_Latino \\\\ \n",
    "      & + \\gamma_{5} Obama\\_Margin\\_2012 + \\phi BlockID + \\epsilon_{i} \\\\ \n",
    "Y_{i} = \\beta_{0} &+ \\beta_{1} Latino + \\beta_{2} Black + \\beta_{3} Arab + \\gamma_{1}Population\\_Density + \\gamma_{2}SES\\_Indicator \\\\\n",
    "      & + \\gamma_{3} Pct\\_Black + \\gamma_{4}Pct\\_Latino + \\gamma_{5} Obama\\_Margin\\_2012 + \\phi BlockID + \\epsilon_{i} \\\\ \n",
    "\\end{align} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=======================================================\n",
      "                            Dependent variable:        \n",
      "                    -----------------------------------\n",
      "                                GotResponse            \n",
      "                           (1)               (2)       \n",
      "-------------------------------------------------------\n",
      "Minority                -0.045***                      \n",
      "                         (0.013)                       \n",
      "                                                       \n",
      "Latino                                     -0.028*     \n",
      "                                           (0.016)     \n",
      "                                                       \n",
      "Black                                       0.004      \n",
      "                                           (0.016)     \n",
      "                                                       \n",
      "Arab                                      -0.110***    \n",
      "                                           (0.016)     \n",
      "                                                       \n",
      "Pop. Density            0.152***          0.149***     \n",
      "                         (0.028)           (0.028)     \n",
      "                                                       \n",
      "Poverty Rate             -0.142            -0.109      \n",
      "                         (0.753)           (0.729)     \n",
      "                                                       \n",
      "Percent Pop. Black      -0.498**          -0.483**     \n",
      "                         (0.231)           (0.235)     \n",
      "                                                       \n",
      "Percent Pop. Latino      -0.355*           -0.367*     \n",
      "                         (0.201)           (0.198)     \n",
      "                                                       \n",
      "2012 Obama Margin        -0.006            -0.015      \n",
      "                         (0.081)           (0.080)     \n",
      "                                                       \n",
      "-------------------------------------------------------\n",
      "Observations              6,439             6,439      \n",
      "R2                        0.334             0.341      \n",
      "Adjusted R2               0.109             0.118      \n",
      "Residual Std. Error 0.466 (df = 4810) 0.464 (df = 4808)\n",
      "=======================================================\n",
      "Note:                       *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "minority_mod2 <- d[ , felm(GotResponse ~ I(ethnic_cue != 0) + dens_log + \n",
    "                           pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                           obama_margin | factor(blockID)) ]\n",
    "minorities_mod2 <- d[ , felm(GotResponse ~ factor(ethnic_cue) + dens_log + \n",
    "                             pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                             obama_margin| factor(blockID)) ]\n",
    "\n",
    "stargazer(minority_mod2, minorities_mod2, \n",
    "          type = \"text\", \n",
    "          se = list(felm_rses(minority_mod2), \n",
    "                    felm_rses(minorities_mod2)), \n",
    "          covariate.labels = c(\"Minority\", \"Latino\", \"Black\", \"Arab\", \n",
    "                               \"Pop. Density\", \"Poverty Rate\", \"Percent Pop. Black\", \n",
    "                               \"Percent Pop. Latino\", \"2012 Obama Margin\") \n",
    "         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"precisionWeighted\"></a>\n",
    "## 2.E Precision-Weighted Treatment Effect Estimates \n",
    "[toc](#toc)\n",
    "\n",
    "In this section we reproduce the main result from White, Nathan and Faller (2015): The difference in the rates of reply to Latino purpotedly registered voters viz white registered voters. To do so, we load the White, Nathan and Faller replication data, execute a limited amount of the replication code, and store results in a model object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cleaning data as do White, Nathan and Faller. \n",
      "* note the bad data.table ethos to suppress printing for jupyter notebook. \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> WNF, Overall Response Rate: </b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Response Rate</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.677</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       " Response Rate\\\\\n",
       "\\hline\n",
       "\t 0.677\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Response Rate | \n",
       "|---|\n",
       "| 0.677 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Response Rate\n",
       "1 0.677        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b> WNF, Response Rate by Condition: </b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Minority</th><th scope=col>Response Rate</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>No   </td><td>0.705</td></tr>\n",
       "\t<tr><td>Yes  </td><td>0.648</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Minority & Response Rate\\\\\n",
       "\\hline\n",
       "\t No    & 0.705\\\\\n",
       "\t Yes   & 0.648\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Minority | Response Rate | \n",
       "|---|---|\n",
       "| No    | 0.705 | \n",
       "| Yes   | 0.648 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Minority Response Rate\n",
       "1 No       0.705        \n",
       "2 Yes      0.648        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rep.d = fread('./data/whiteNathanFaller_replicationFiles/voterIDexp_data_sept2014.csv')\n",
    "\n",
    "## clean data \n",
    "cat('cleaning data as do White, Nathan and Faller. \\n')\n",
    "cat('* note the bad data.table ethos to suppress printing for jupyter notebook. \\n')\n",
    "rep.d <- rep.d[ , abs_acc := 0]\n",
    "rep.d <- rep.d[accuracy==\"Absolutely accurate\", abs_acc := 1]\n",
    "rep.d <- rep.d[is.na(response)==TRUE|response==0, abs_acc := NA]\n",
    "rep.d <- rep.d[ , non_info := 0]\n",
    "rep.d <- rep.d[accuracy==\"Non-informative response\", non_info := 1]\n",
    "rep.d <- rep.d[is.na(response)|response==0, non_info := NA]\n",
    "\n",
    "## create coding for vra_all \n",
    "rep.d <- rep.d[ , vra_all := 0]\n",
    "rep.d <- rep.d[(preclear_states==1|spanishcov==1)&is.na(preclear_states==1)==FALSE &is.na(spanishcov)==FALSE, vra_all := 1]\n",
    "\n",
    "## split to just the first mailing (reported in main text of White, Nathan,\n",
    "## Faller 2015\n",
    "week1 <- rep.d[week==1]\n",
    "week1 <- week1[code !=41 & code!=12]\n",
    "week1 <- week1[ , code.fac := as.factor(as.character(code))]\n",
    "\n",
    "## split to form and 'id' dataset and a 'control' dataset \n",
    "week1id <- week1[week1$first_text==1]\n",
    "week1pr <- week1[week1$first_text==0]\n",
    "\n",
    "display_html(\"<b> WNF, Overall Response Rate: </b>\")\n",
    "week1id[ , .(\"Response Rate\" = mean(response, na.rm = T))]\n",
    "\n",
    "display_html(\"<b> WNF, Response Rate by Condition: </b>\")\n",
    "week1id[ , .(meanResponse = mean(response, na.rm = T)), \n",
    "       keyby = .(\"Minority\" = first_name_latino)] %>% \n",
    "  .[ , .(\"Minority\"  = c(\"No\", \"Yes\"), \n",
    "         \"Response Rate\" = meanResponse)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Green and Gerber ATE and SE(ATE) \n",
    "In the following cell, we calculate the average treatment effect and standard error of that average treatment effect as presented in Green and Gerbers' _Field Experiemnts_. Additionally, we use fixed-effects meta analysis, to combine the results of our study with the results "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## calcuate wnf latino effect\n",
    "diff.wnf = week1id[ , .(mean.response = mean(response, na.rm=T)),\n",
    "    keyby = .(first_name_latino)][ , diff(mean.response)]\n",
    "se.wnf = week1id[ , .(var.part = var(response, na.rm=T), n = .N),\n",
    "    keyby = .(first_name_latino)][ ,\n",
    "        .(gg.se = sqrt(sum(var.part / n)))]\n",
    "confint.wnf = diff.wnf + qt(p=c(.025, .975), df = week1id[ , .N] - 4) %o% se.wnf\n",
    "\n",
    "## calculate the 2016 latino effect \n",
    "diff.h.2016 = d[ethnic_cue %in% 0:1, .(mean.response = mean(GotResponse, na.rm = T)),\n",
    "    keyby = .(ethnic_cue == 1)][ , diff(mean.response)]\n",
    "se.h.2016 = d[ethnic_cue %in% 0:1, .(var.part = var(GotResponse, na.rm=T), n = .N),\n",
    "    keyby = .(ethnic_cue == 1)][ ,\n",
    "                  .(gg.se = sqrt(sum(var.part / n)))]\n",
    "confint.h.2016 = diff.h.2016 +\n",
    "    qt(p=c(.025, .975), df=d[ethnic_cue %in% 0:1, .N] - 4) %o% se.h.2016\n",
    "\n",
    "## calculate effect of all minoritites vs. white in 2016 experiment\n",
    "diff.2016 = d[ , .(mean.response = mean(GotResponse, na.rm = T)),\n",
    "    keyby = .(ethnic_cue != 0)][ , diff(mean.response)]\n",
    "se.2016 = d[ , .(var.part = var(GotResponse, na.rm=T), n = .N),\n",
    "    keyby = .(ethnic_cue != 0)][ ,\n",
    "                  .(gg.se = sqrt(sum(var.part / n)))]\n",
    "confint.2016 = diff.2016 +\n",
    "    qt(p=c(.025, .975), df=d[, .N] - 7) %o% se.2016\n",
    "\n",
    "diffs.2016 = d[ , .(mean.response = mean(GotResponse, na.rm = T), n = .N),\n",
    "    keyby = .(ethnic_cue)][ , .(mean.response = mean.response,\n",
    "        effect = mean.response - mean.response[1], n = n)\n",
    "        ]\n",
    "ses.2016 = d[ , .(var.part = var(GotResponse, na.rm=T), n = .N),\n",
    "    keyby = .(ethnic_cue)][ ,\n",
    "        .(gg.se = sqrt(sum(var.part[1] / n[1], var.part / n))), keyby = .(ethnic_cue)\n",
    "        ]\n",
    "diffs.2016 = cbind(diffs.2016, ses.2016)\n",
    "\n",
    "diffs.2016[ , ':='(low.ci  = effect - qt(0.025, df = n[1] + n-4) * gg.se, \n",
    "                   high.ci = effect + qt(0.025, df = n[1] + n-4) * gg.se) ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is the precision weighted ATE? \n",
    "In the following cells, we compute the precision weighted ATE drawn from *FEDAI*. In particular, we estimate that the pooled ATE for the prior and current experiments, $\\hat{ATE}_{pooled}$ is: \n",
    "\n",
    "\\begin{equation} \n",
    "    \\hat{ATE}_{pooled} = \\frac{\\frac{1}{\\hat{\\sigma}^{2}_{1}}}{\\frac{1}{\\hat{\\sigma}^{2}_{1}} + \\frac{1}{\\hat{\\sigma}^{2}_{2}}} \\hat{ATE}_{1} + \\frac{\\frac{1}{\\hat{\\sigma}^{2}_{2}}}{\\frac{1}{\\hat{\\sigma}^{2}_{1}} + \\frac{1}{\\hat{\\sigma}^{2}_{2}}} \\hat{ATE}_{2}\n",
    "\\end{equation} \n",
    "\n",
    "and the standard error for this pooled estimator is: \n",
    "\n",
    "\\begin{equation} \n",
    "    \\sqrt{Var(\\hat{ATE}_{pooled})} = \\left[\\frac{1}{\\hat{\\sigma}^{2}_{1}} + \\frac{1}{\\hat{\\sigma}^{2}_{2}}\\right]^{-0.5}\n",
    "\\end{equation} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "pw.ate = gg.pooled.ate(ate1=diff.wnf, ate2=diff.h.2016, se1=se.wnf, se2=se.h.2016)\n",
    "pw.se = gg.pooled.se(se1=se.wnf, se2=se.h.2016)\n",
    "pw.ci = as.numeric(pw.ate) + qt(c(.025, .975), sum(d[ethnic_cue %in% 0:1, .N], week1[, .N])) %o% as.numeric(pw.se)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'The precision weighted ATE is: -4.41 percent.'"
      ],
      "text/latex": [
       "'The precision weighted ATE is: -4.41 percent.'"
      ],
      "text/markdown": [
       "'The precision weighted ATE is: -4.41 percent.'"
      ],
      "text/plain": [
       "[1] \"The precision weighted ATE is: -4.41 percent.\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'The precision weighted SE is: 1.18 percent.'"
      ],
      "text/latex": [
       "'The precision weighted SE is: 1.18 percent.'"
      ],
      "text/markdown": [
       "'The precision weighted SE is: 1.18 percent.'"
      ],
      "text/plain": [
       "[1] \"The precision weighted SE is: 1.18 percent.\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sprintf(\"The precision weighted ATE is: %.2f percent.\", pw.ate*100)\n",
    "sprintf(\"The precision weighted SE is: %.2f percent.\", pw.se * 100)\n",
    "# sprintf(\"The Wald test statistic is: %.2f, and the p-value for that statistic under a normal distribution is: %.4f\", pw.ate / pw.se, pnorm(q = pw.ate/pw.se))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"precisionWeightedPlot\"></a>\n",
    "## Plot these Effects \n",
    "[toc](#toc)\n",
    "\n",
    "In this section, we create the data objects used in the plot in *Figure 1* in the main text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot.obj = data.table(\n",
    "    model = c(\"WNF\", \"2016 Minority\", \"2016 Latino\", \n",
    "              \"2016 Black\", \"2016 Arab\", \"Precision Weighted\"), \n",
    "    effect = NA, se = NA, \n",
    "    low.ci = NA, high.ci = NA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>model</th><th scope=col>effect</th><th scope=col>se</th><th scope=col>low.ci</th><th scope=col>high.ci</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>WNF               </td><td>-0.0568           </td><td>0.0162            </td><td>-0.0885           </td><td>-0.0250           </td></tr>\n",
       "\t<tr><td>2016 Latino       </td><td>-0.0297           </td><td>0.0174            </td><td> 0.0044           </td><td>-0.0638           </td></tr>\n",
       "\t<tr><td>Precision Weighted</td><td>-0.0441           </td><td>0.0118            </td><td>-0.0673           </td><td>-0.0209           </td></tr>\n",
       "\t<tr><td>2016 Minority     </td><td>-0.047            </td><td>0.0141            </td><td>-0.0746           </td><td>-0.0194           </td></tr>\n",
       "\t<tr><td>2016 Black        </td><td>0.0011            </td><td>0.0171            </td><td> 0.0347           </td><td>-0.0325           </td></tr>\n",
       "\t<tr><td>2016 Arab         </td><td>-0.113            </td><td>0.0177            </td><td>-0.0781           </td><td>-0.1473           </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       " model & effect & se & low.ci & high.ci\\\\\n",
       "\\hline\n",
       "\t WNF                & -0.0568            & 0.0162             & -0.0885            & -0.0250           \\\\\n",
       "\t 2016 Latino        & -0.0297            & 0.0174             &  0.0044            & -0.0638           \\\\\n",
       "\t Precision Weighted & -0.0441            & 0.0118             & -0.0673            & -0.0209           \\\\\n",
       "\t 2016 Minority      & -0.047             & 0.0141             & -0.0746            & -0.0194           \\\\\n",
       "\t 2016 Black         & 0.0011             & 0.0171             &  0.0347            & -0.0325           \\\\\n",
       "\t 2016 Arab          & -0.113             & 0.0177             & -0.0781            & -0.1473           \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "model | effect | se | low.ci | high.ci | \n",
       "|---|---|---|---|---|---|\n",
       "| WNF                | -0.0568            | 0.0162             | -0.0885            | -0.0250            | \n",
       "| 2016 Latino        | -0.0297            | 0.0174             |  0.0044            | -0.0638            | \n",
       "| Precision Weighted | -0.0441            | 0.0118             | -0.0673            | -0.0209            | \n",
       "| 2016 Minority      | -0.047             | 0.0141             | -0.0746            | -0.0194            | \n",
       "| 2016 Black         | 0.0011             | 0.0171             |  0.0347            | -0.0325            | \n",
       "| 2016 Arab          | -0.113             | 0.0177             | -0.0781            | -0.1473            | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  model              effect  se     low.ci  high.ci\n",
       "1 WNF                -0.0568 0.0162 -0.0885 -0.0250\n",
       "2 2016 Latino        -0.0297 0.0174  0.0044 -0.0638\n",
       "3 Precision Weighted -0.0441 0.0118 -0.0673 -0.0209\n",
       "4 2016 Minority      -0.047  0.0141 -0.0746 -0.0194\n",
       "5 2016 Black         0.0011  0.0171  0.0347 -0.0325\n",
       "6 2016 Arab          -0.113  0.0177 -0.0781 -0.1473"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot.obj[ , ':='(effect = c(diff.wnf, diff.2016, diffs.2016[-1, effect], pw.ate), \n",
    "                 se = c(se.wnf, se.2016, diffs.2016[-1, gg.se], pw.se), \n",
    "                 low.ci = c(confint.wnf[1], confint.2016[1], diffs.2016[-1, low.ci], pw.ci[1]),\n",
    "                 high.ci = c(confint.wnf[2], confint.2016[2], diffs.2016[-1, high.ci], pw.ci[2]))]\n",
    "plot.obj = plot.obj[c(1, 3, 6, 2,4,5)]\n",
    "plot.obj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pdf(file = './figures-tables/fig1.pdf', width = 14, height = 5)\n",
    "\n",
    "par(lwd = 2,             cex.main = 2,\n",
    "    font.lab = 2,        cex.lab = 2,\n",
    "    font.lab = 2,        cex.axis = 1.5,\n",
    "    font.axis = 2,       font = 2,\n",
    "    col.axis = \"gray50\", las = 1,\n",
    "    mar = c(5.1, 5.1, 4.1, 0.5),\n",
    "    bty = \"n\",           pch = 19\n",
    "    )\n",
    "\n",
    "layout(mat=matrix(1:2, byrow=T, nrow=1, ncol=2),\n",
    "       widths=c(1,1),\n",
    "       heights=c(1,1)\n",
    "       )\n",
    "\n",
    "plot.obj[model %in% c(\"2016 Latino\", \"2016 Black\", \"2016 Arab\") , \n",
    "         plot(x = 1:.N, y = effect, ylim = c(-.15, .05), xlim = c(0.5, 3.5), \n",
    "              xaxt = 'n', xlab = NA, yaxt = 'n', ylab = NA, \n",
    "              main = \"2016 Election Official Bias\", \n",
    "              cex = 2)]\n",
    "axis(1, at = 1:3, labels = c(\"Latino\", \"Black\", \"Arab\"), lwd = 0)\n",
    "axis(2, at = seq(-.15, .05, by = .05), labels = paste0(seq(-15,5,5), \"%\"))\n",
    "# text(x = .75, y = 0.04, labels = \"(a)\", cex = 2, col = \"Grey 50\")\n",
    "abline(h=0, col = \"grey\", lty = 2)\n",
    "for(i in 1:3) { \n",
    "    plot.obj[model %in% c(\"2016 Latino\", \"2016 Black\", \"2016 Arab\")[i], \n",
    "             arrows(x0=i, x1=i, y0 = low.ci, y1 = high.ci, code = 0, lwd = 4)]\n",
    "}\n",
    "for(i in 1:3) { \n",
    "    plot.obj[model %in% c(\"2016 Latino\", \"2016 Black\", \"2016 Arab\")[i], \n",
    "             arrows(x0=i, x1=i, y0 = as.numeric(effect)-as.numeric(se), y1 = as.numeric(effect)+as.numeric(se), code = 0, lwd = 8)]\n",
    "}\n",
    "\n",
    "plot.obj[model %in% c(\"WNF\", \"2016 Latino\", \"Precision Weighted\") , \n",
    "         plot(x = 1:.N, y = effect, ylim = c(-.15, .05), xlim = c(0.5, 3.5), \n",
    "              xaxt = 'n', xlab = NA, yaxt = 'n', ylab = NA, \n",
    "              main = \"2012 & 2016 Election Official Bias\", \n",
    "              cex = 2)]\n",
    "axis(1, at = 1:3, labels = c(\"2012 Latino\", \"2016 Latino\", \"Weighted \\nAverage\"), lwd = 0)\n",
    "axis(2, at = seq(-.15, .05, by = .05), labels = paste0(seq(-15,5,5), \"%\"))\n",
    "abline(h=0, col = \"grey\", lty = 2)\n",
    "# text(x = .75, y = 0.04, labels = \"(b)\", cex = 2, col = \"Grey 50\")\n",
    "for(i in 1:3) { \n",
    "    plot.obj[model %in% c(\"WNF\", \"2016 Latino\", \"Precision Weighted\")[i], \n",
    "             arrows(x0=i, x1=i, y0 = low.ci, y1 = high.ci, code = 0, lwd = 4)]\n",
    "}\n",
    "for(i in 1:3) { \n",
    "    plot.obj[model %in% c(\"WNF\", \"2016 Latino\", \"Precision Weighted\")[i], \n",
    "             arrows(x0=i, x1=i, y0 = as.numeric(effect)-as.numeric(se), y1 = as.numeric(effect)+as.numeric(se), code = 0, lwd = 8)]\n",
    "}\n",
    "\n",
    "dev.off()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../figures-tables/fig1.pdf\" alt=\"Experiment Overview\" width=500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"hte\"></a>\n",
    "# 3. Heterogeneous Treatment Effects \n",
    "\n",
    "<a id=\"demos\"></a>\n",
    "## 3.A Demographic Characteristics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently, we do not build this data using the the data intake pipeline. This allows for matching at the county and sub-county level, as was performed for all other variables. We note, here, that there is one strange problem that we are working to identify: there are 33 registrars that don't match back onto the original dataset. @diana is doing the QA to see what is going on. Once those are found, we intend to move this late-stage merge into the data-intake pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b> ANOVA test of Black HTE interaction. </b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Res.Df</th><th scope=col>RSS</th><th scope=col>Df</th><th scope=col>Sum of Sq</th><th scope=col>F</th><th scope=col>Pr(&gt;F)</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>6435   </td><td>1564   </td><td>NA     </td><td>     NA</td><td>    NA </td><td>   NA  </td></tr>\n",
       "\t<tr><td>6436   </td><td>1564   </td><td>-1     </td><td>-0.0195</td><td>0.0802 </td><td>0.777  </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " Res.Df & RSS & Df & Sum of Sq & F & Pr(>F)\\\\\n",
       "\\hline\n",
       "\t 6435    & 1564    & NA      &      NA &     NA  &    NA  \\\\\n",
       "\t 6436    & 1564    & -1      & -0.0195 & 0.0802  & 0.777  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Res.Df | RSS | Df | Sum of Sq | F | Pr(>F) | \n",
       "|---|---|\n",
       "| 6435    | 1564    | NA      |      NA |     NA  |    NA   | \n",
       "| 6436    | 1564    | -1      | -0.0195 | 0.0802  | 0.777   | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Res.Df RSS  Df Sum of Sq F      Pr(>F)\n",
       "1 6435   1564 NA      NA       NA    NA \n",
       "2 6436   1564 -1 -0.0195   0.0802 0.777 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hte.black.null = d[ , lm(GotResponse ~ I(ethnic_cue != 0) + pct_race_black)]\n",
    "hte.black      = d[ , lm(GotResponse ~ I(ethnic_cue != 0) * pct_race_black)]\n",
    "\n",
    "display_html(\"<b> ANOVA test of Black HTE interaction. </b>\")\n",
    "anova(hte.black, hte.black.null)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b> ANOVA test of Latino HTE interaction. </b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Res.Df</th><th scope=col>RSS</th><th scope=col>Df</th><th scope=col>Sum of Sq</th><th scope=col>F</th><th scope=col>Pr(&gt;F)</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>6435  </td><td>1565  </td><td>NA    </td><td>    NA</td><td> NA   </td><td>  NA  </td></tr>\n",
       "\t<tr><td>6436  </td><td>1565  </td><td>-1    </td><td>-0.122</td><td>0.5   </td><td>0.48  </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " Res.Df & RSS & Df & Sum of Sq & F & Pr(>F)\\\\\n",
       "\\hline\n",
       "\t 6435   & 1565   & NA     &     NA &  NA    &   NA  \\\\\n",
       "\t 6436   & 1565   & -1     & -0.122 & 0.5    & 0.48  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Res.Df | RSS | Df | Sum of Sq | F | Pr(>F) | \n",
       "|---|---|\n",
       "| 6435   | 1565   | NA     |     NA |  NA    |   NA   | \n",
       "| 6436   | 1565   | -1     | -0.122 | 0.5    | 0.48   | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Res.Df RSS  Df Sum of Sq F   Pr(>F)\n",
       "1 6435   1565 NA     NA     NA   NA  \n",
       "2 6436   1565 -1 -0.122    0.5 0.48  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hte.latino.null = d[ , lm(GotResponse ~ I(ethnic_cue != 0) + pct_latino)]\n",
    "hte.latino      = d[ , lm(GotResponse ~ I(ethnic_cue != 0) * pct_latino)]\n",
    "\n",
    "display_html(\"<b> ANOVA test of Latino HTE interaction. </b>\")\n",
    "anova(hte.latino, hte.latino.null)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b> ANOVA test of Arab HTE interaction. </b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Res.Df</th><th scope=col>RSS</th><th scope=col>Df</th><th scope=col>Sum of Sq</th><th scope=col>F</th><th scope=col>Pr(&gt;F)</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>6402   </td><td>1560   </td><td>NA     </td><td>     NA</td><td>   NA  </td><td>   NA  </td></tr>\n",
       "\t<tr><td>6403   </td><td>1560   </td><td>-1     </td><td>-0.0546</td><td>0.224  </td><td>0.636  </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " Res.Df & RSS & Df & Sum of Sq & F & Pr(>F)\\\\\n",
       "\\hline\n",
       "\t 6402    & 1560    & NA      &      NA &    NA   &    NA  \\\\\n",
       "\t 6403    & 1560    & -1      & -0.0546 & 0.224   & 0.636  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Res.Df | RSS | Df | Sum of Sq | F | Pr(>F) | \n",
       "|---|---|\n",
       "| 6402    | 1560    | NA      |      NA |    NA   |    NA   | \n",
       "| 6403    | 1560    | -1      | -0.0546 | 0.224   | 0.636   | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Res.Df RSS  Df Sum of Sq F     Pr(>F)\n",
       "1 6402   1560 NA      NA      NA    NA \n",
       "2 6403   1560 -1 -0.0546   0.224 0.636 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hte.arab.null = d[ , lm(GotResponse ~ I(ethnic_cue != 0) + pct_arab)]\n",
    "hte.arab      = d[ , lm(GotResponse ~ I(ethnic_cue != 0) * pct_arab)]\n",
    "\n",
    "display_html(\"<b> ANOVA test of Arab HTE interaction. </b>\")\n",
    "anova(hte.arab, hte.arab.null)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Block Fixed Effects \n",
    "Does the presence of minority groups in a district associated with differential responsiveness on the part of registrars? To estimate this effect, we fit models that include block fixed effects together with an interaction term for the percent of a geography's population that is composed of the a particular minority class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "hte_felm_dist_null_latino = d[ , felm(GotResponse ~ I(ethnic_cue != 0) + pct_latino     | factor(blockID))]\n",
    "hte_felm_dist_null_black  = d[ , felm(GotResponse ~ I(ethnic_cue != 0) + pct_race_black | factor(blockID))]\n",
    "hte_felm_dist_null_arab   = d[ , felm(GotResponse ~ I(ethnic_cue != 0) + pct_arab       | factor(blockID))]\n",
    "hte_felm_dist_latino      = d[ , felm(GotResponse ~ I(ethnic_cue != 0) * pct_latino     | factor(blockID))]\n",
    "hte_felm_dist_black       = d[ , felm(GotResponse ~ I(ethnic_cue != 0) * pct_race_black | factor(blockID))]\n",
    "hte_felm_dist_arab        = d[ , felm(GotResponse ~ I(ethnic_cue != 0) * pct_arab       | factor(blockID))]\n",
    "hte_felm_dist_null_latino1= d[ , felm(GotResponse ~ factor(ethnic_cue) + pct_latino     | factor(blockID))]\n",
    "hte_felm_dist_null_black1 = d[ , felm(GotResponse ~ factor(ethnic_cue) + pct_race_black | factor(blockID))]\n",
    "hte_felm_dist_null_arab1  = d[ , felm(GotResponse ~ factor(ethnic_cue) + pct_arab       | factor(blockID))]\n",
    "hte_felm_dist_latino1     = d[ , felm(GotResponse ~ factor(ethnic_cue) * pct_latino     | factor(blockID))]\n",
    "hte_felm_dist_black1      = d[ , felm(GotResponse ~ factor(ethnic_cue) * pct_race_black | factor(blockID))]\n",
    "hte_felm_dist_arab1       = d[ , felm(GotResponse ~ factor(ethnic_cue) * pct_arab       | factor(blockID))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no evidence that district non-white composition has any moderating effect on the discrimination against all minority groups. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================================================================\n",
      "                                               Dependent variable:                 \n",
      "                              -----------------------------------------------------\n",
      "                                                   GotResponse                     \n",
      "                                     (1)               (2)               (3)       \n",
      "-----------------------------------------------------------------------------------\n",
      "Minority Cue                      -0.052***         -0.048***         -0.044***    \n",
      "                                   (0.015)           (0.015)           (0.015)     \n",
      "                                                                                   \n",
      "Percent Latino                     -0.241                                          \n",
      "                                   (0.236)                                         \n",
      "                                                                                   \n",
      "Percent Latino * Minority Cue       0.093                                          \n",
      "                                   (0.143)                                         \n",
      "                                                                                   \n",
      "Percent Black                                        -0.163                        \n",
      "                                                     (0.230)                       \n",
      "                                                                                   \n",
      "Percent Black * Minority Cue                          0.013                        \n",
      "                                                     (0.133)                       \n",
      "                                                                                   \n",
      "Percent Arab                                                            1.580      \n",
      "                                                                       (2.440)     \n",
      "                                                                                   \n",
      "Percent Arab * Minority Cue                                            -1.270      \n",
      "                                                                       (2.530)     \n",
      "                                                                                   \n",
      "-----------------------------------------------------------------------------------\n",
      "Observations                        6,439             6,439             6,406      \n",
      "R2                                  0.330             0.330             0.329      \n",
      "Adjusted R2                         0.104             0.103             0.101      \n",
      "Residual Std. Error           0.468 (df = 4813) 0.468 (df = 4813) 0.468 (df = 4780)\n",
      "===================================================================================\n",
      "Note:                                                   *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "stargazer(hte_felm_dist_latino, hte_felm_dist_black, hte_felm_dist_arab, type= \"text\", \n",
    "          se = list(felm_rses(hte_felm_dist_latino), felm_rses(hte_felm_dist_black), felm_rses(hte_felm_dist_arab)),\n",
    "          covariate.labels = c(\"Minority Cue\", \n",
    "                               \"Percent Latino\", \"Percent Latino * Minority Cue\", \n",
    "                               \"Percent Black\",  \"Percent Black * Minority Cue\", \n",
    "                               \"Percent Arab\",   \"Percent Arab * Minority Cue\"), \n",
    "                   omit.stat = c(\"F\", \"ser\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = capture.output( \n",
    "    stargazer(hte_felm_dist_latino, hte_felm_dist_black, hte_felm_dist_arab, \n",
    "          type= \"latex\", \n",
    "          out = \"./figures-tables/districtHTEminorityTable.tex\", \n",
    "          label = \"tab:districtHTE\",\n",
    "          se = list(felm_rses(hte_felm_dist_latino), felm_rses(hte_felm_dist_black), felm_rses(hte_felm_dist_arab)), \n",
    "          covariate.labels = c(\"Minority\", \"Percent Latino\", \"Percent Latino $\\\\times$ Minority\", \n",
    "                               \"Percent Black\", \"Percent Black $\\\\times$ Minority\", \n",
    "                               \"Percent Arab\", \"Percent Arab $\\\\times$ Minority\"), \n",
    "          omit.stat = c(\"F\", \"ser\"), \n",
    "          align = TRUE    \n",
    "          )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is evidence here that the effect of the Latino cue is moderated by the percent of Latino population in a geography. There is no such support for percent of black population living in a geography, nor for percent of arab population living in a geography. \n",
    "\n",
    "We note that there are a small number of geographies -- 33 in total, representing 0.005 the sample -- for which we were not able to calculate the percent of Arab American residents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "                                 Dependent variable:     \n",
      "                            -----------------------------\n",
      "                                     GotResponse         \n",
      "                               (1)       (2)       (3)   \n",
      "---------------------------------------------------------\n",
      "Latino                      -0.049***  -0.026    -0.028  \n",
      "                             (0.019)   (0.018)   (0.018) \n",
      "                                                         \n",
      "Black                         0.013    -0.003     0.003  \n",
      "                             (0.019)   (0.018)   (0.018) \n",
      "                                                         \n",
      "Arab                        -0.121*** -0.113*** -0.109***\n",
      "                             (0.019)   (0.018)   (0.018) \n",
      "                                                         \n",
      "Percent Latino               -0.227                      \n",
      "                             (0.233)                     \n",
      "                                                         \n",
      "Percent Latino * Latino Cue  0.345**                     \n",
      "                             (0.167)                     \n",
      "                                                         \n",
      "Percent Latino * Black Cue   -0.199                      \n",
      "                             (0.174)                     \n",
      "                                                         \n",
      "Percent Latino * Arab Cue     0.138                      \n",
      "                             (0.168)                     \n",
      "                                                         \n",
      "Percent Black                          -0.173            \n",
      "                                       (0.234)           \n",
      "                                                         \n",
      "Percent Black * Latino Cue             -0.098            \n",
      "                                       (0.162)           \n",
      "                                                         \n",
      "Percent Black * Black Cue               0.119            \n",
      "                                       (0.166)           \n",
      "                                                         \n",
      "Percent Black * Arab Cue                0.008            \n",
      "                                       (0.156)           \n",
      "                                                         \n",
      "Percent Arab                                      1.680  \n",
      "                                                 (2.460) \n",
      "                                                         \n",
      "Percent Arab * Latino Cue                        -0.850  \n",
      "                                                 (2.780) \n",
      "                                                         \n",
      "Percent Arab * Black Cue                         -0.657  \n",
      "                                                 (2.770) \n",
      "                                                         \n",
      "Percent Arab * Arab Cue                          -1.740  \n",
      "                                                 (2.670) \n",
      "                                                         \n",
      "---------------------------------------------------------\n",
      "Observations                  6,439     6,439     6,406  \n",
      "R2                            0.339     0.337     0.337  \n",
      "Adjusted R2                   0.115     0.113     0.110  \n",
      "=========================================================\n",
      "Note:                         *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "stargazer(hte_felm_dist_latino1, hte_felm_dist_black1, hte_felm_dist_arab1, type = \"text\",\n",
    "          se = list(felm_rses(hte_felm_dist_latino1), felm_rses(hte_felm_dist_black1), felm_rses(hte_felm_dist_arab1)), \n",
    "          covariate.labels = c(\"Latino\", \"Black\", \"Arab\", \n",
    "                               \"Percent Latino\", \n",
    "                               \"Percent Latino * Latino Cue\", \"Percent Latino * Black Cue\", \"Percent Latino * Arab Cue\", \n",
    "                               \"Percent Black\", \n",
    "                               \"Percent Black * Latino Cue\", \"Percent Black * Black Cue\", \"Percent Black * Arab Cue\", \n",
    "                               \"Percent Arab\", \n",
    "                               \"Percent Arab * Latino Cue\", \"Percent Arab * Black Cue\", \"Percent Arab * Arab Cue\"), \n",
    "          omit.stat = c(\"F\", \"ser\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = capture.output(\n",
    "    stargazer(hte_felm_dist_latino1, hte_felm_dist_black1, hte_felm_dist_arab1, \n",
    "          type = \"latex\", \n",
    "          out = \"./figures-tables/districtHTEminorityTable1.tex\", \n",
    "          label = \"tab:districtHTE1\",\n",
    "          se = list(felm_rses(hte_felm_dist_latino1), felm_rses(hte_felm_dist_black1), felm_rses(hte_felm_dist_arab1)), \n",
    "          covariate.labels = c(\"Latino\", \"Black\", \"Arab\", \"Percent Latino\",\n",
    "                               \"Percent Latino $\\\\times$ Latino\", \"Percent Latino $\\\\times$ Black\", \"Percent Latino $\\\\times$ Arab\", \n",
    "                               \"Percent Black\", \n",
    "                               \"Percent Black  $\\\\times$ Latino\", \"Percent Black  $\\\\times$ Black\", \"Percent Black  $\\\\times$ Arab\", \n",
    "                               \"Percent Arab\", \n",
    "                               \"Percent Arab   $\\\\times$ Latino\", \"Percent Arab   $\\\\times$ Black\", \"Percent Arab   $\\\\times$ Arab\"), \n",
    "              add.lines = list(c(\"Block FE\", \"Yes\", \"Yes\", \"Yes\")), \n",
    "          omit.stat = c(\"F\", \"ser\", \"adj.rsq\"), \n",
    "          align = T, \n",
    "          font.size = \"footnotesize\"    \n",
    "          )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We mean impute these values for these missing `pct_arab` to check that results do not change with vs. without the inclusion of these missing values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "                                 Dependent variable:     \n",
      "                            -----------------------------\n",
      "                                     GotResponse         \n",
      "                               (1)       (2)       (3)   \n",
      "---------------------------------------------------------\n",
      "Latino                      -0.049***  -0.026    -0.027  \n",
      "                             (0.019)   (0.018)   (0.018) \n",
      "                                                         \n",
      "Black                         0.013    -0.003     0.004  \n",
      "                             (0.019)   (0.018)   (0.018) \n",
      "                                                         \n",
      "Arab                        -0.121*** -0.113*** -0.108***\n",
      "                             (0.019)   (0.018)   (0.018) \n",
      "                                                         \n",
      "Percent Latino               -0.227                      \n",
      "                             (0.233)                     \n",
      "                                                         \n",
      "Percent Latino * Latino Cue  0.345**                     \n",
      "                             (0.167)                     \n",
      "                                                         \n",
      "Percent Latino * Black Cue   -0.199                      \n",
      "                             (0.174)                     \n",
      "                                                         \n",
      "Percent Latino * Arab Cue     0.138                      \n",
      "                             (0.168)                     \n",
      "                                                         \n",
      "Percent Black                          -0.173            \n",
      "                                       (0.234)           \n",
      "                                                         \n",
      "Percent Black * Latino Cue             -0.098            \n",
      "                                       (0.162)           \n",
      "                                                         \n",
      "Percent Black * Black Cue               0.119            \n",
      "                                       (0.166)           \n",
      "                                                         \n",
      "Percent Black * Arab Cue                0.008            \n",
      "                                       (0.156)           \n",
      "                                                         \n",
      "Percent Arab                                      1.690  \n",
      "                                                         \n",
      "                                                         \n",
      "Percent Arab * Latino Cue                        -0.872  \n",
      "                                                         \n",
      "                                                         \n",
      "Percent Arab * Black Cue                         -0.682  \n",
      "                                                         \n",
      "                                                         \n",
      "Percent Arab * Arab Cue                          -1.760  \n",
      "                                                         \n",
      "                                                         \n",
      "---------------------------------------------------------\n",
      "Observations                  6,439     6,439     6,439  \n",
      "R2                            0.339     0.337     0.337  \n",
      "Adjusted R2                   0.115     0.113     0.112  \n",
      "=========================================================\n",
      "Note:                         *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "d[ , i_pct_arab := pct_arab]\n",
    "m_arab = d[ , mean(pct_arab, na.rm=T)]\n",
    "d[is.na(pct_arab), i_pct_arab := m_arab]\n",
    "rm(m_arab)\n",
    "\n",
    "hte_felm_dist_null_arab_i   = d[ , felm(GotResponse ~ I(ethnic_cue != 0) + pct_arab         | factor(blockID))]\n",
    "hte_felm_dist_arab_i        = d[ , felm(GotResponse ~ I(ethnic_cue != 0) * pct_arab         | factor(blockID))]\n",
    "hte_felm_dist_arab1_i       = d[ , felm(GotResponse ~ factor(ethnic_cue) * i_pct_arab       | factor(blockID))]\n",
    "\n",
    "stargazer(hte_felm_dist_latino1, hte_felm_dist_black1, hte_felm_dist_arab1_i, type = \"text\",\n",
    "          se = list(felm_rses(hte_felm_dist_latino1), felm_rses(hte_felm_dist_black1), felm_rses(hte_felm_dist_arab1)), \n",
    "          covariate.labels = c(\"Latino\", \"Black\", \"Arab\", \n",
    "                               \"Percent Latino\", \n",
    "                               \"Percent Latino * Latino Cue\", \"Percent Latino * Black Cue\", \"Percent Latino * Arab Cue\", \n",
    "                               \"Percent Black\", \n",
    "                               \"Percent Black * Latino Cue\", \"Percent Black * Black Cue\", \"Percent Black * Arab Cue\", \n",
    "                               \"Percent Arab\", \n",
    "                               \"Percent Arab * Latino Cue\", \"Percent Arab * Black Cue\", \"Percent Arab * Arab Cue\"), \n",
    "          omit.stat = c(\"F\", \"ser\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, the distribution of Arab-American individuals is unique viz the settlement patters of other groups. Whereas there are very few geographies that have either zero Latino or zero Black population, geographies that have zero arab-american settlement are relatively more common. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Zero Arab American</th><th scope=col>Zero Black American</th><th scope=col>Zero Latino</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.495 </td><td>0.265 </td><td>0.0848</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " Zero Arab American & Zero Black American & Zero Latino\\\\\n",
       "\\hline\n",
       "\t 0.495  & 0.265  & 0.0848\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Zero Arab American | Zero Black American | Zero Latino | \n",
       "|---|\n",
       "| 0.495  | 0.265  | 0.0848 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Zero Arab American Zero Black American Zero Latino\n",
       "1 0.495              0.265               0.0848     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d[ , .(\"Zero Arab American\" = mean(pct_arab == 0, na.rm=T),\n",
    "       \"Zero Black American\" = mean(pct_race_black == 0, na.rm=T),\n",
    "       \"Zero Latino\" = mean(pct_latino == 0, na.rm=T))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because distributions of people were determined pre-treatment, we can in princple condition on this variable. Despite the distribution of peoples, we still have good randomization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>White Cue</th><th scope=col>Latino Cue</th><th scope=col>Black Cue</th><th scope=col>Arab Cue</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>834</td><td>789</td><td>812</td><td>802</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " White Cue & Latino Cue & Black Cue & Arab Cue\\\\\n",
       "\\hline\n",
       "\t 834 & 789 & 812 & 802\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "White Cue | Latino Cue | Black Cue | Arab Cue | \n",
       "|---|\n",
       "| 834 | 789 | 812 | 802 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  White Cue Latino Cue Black Cue Arab Cue\n",
       "1 834       789        812       802     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d %>% \n",
    "  .[pct_arab > 0, .(\"White Cue\"  = sum(ethnic_cue == 0), \n",
    "                    \"Latino Cue\" = sum(ethnic_cue == 1), \n",
    "                    \"Black Cue\"  = sum(ethnic_cue == 2), \n",
    "                    \"Arab Cue\"   = sum(ethnic_cue == 3))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And so, to assess whether the distribution of Arab American individuals conditions the response of local election officials, we perform a rescaling of the `pct_arab` variable. In particular, we create a three-level indicator varaible for the percent of arab population. If there are zero individuals of Arab heritage, we code `median_arab_pop == 0`. Within the geographies that have *some* Arab heritage population, we perform a median split, coding `median_arab_pop == 1` if the geography falls lower than the median split and `median_arab_pop == 2` if the geography falls higher than the median split. \n",
    "\n",
    "Coincedentally, rough 50% of US geographies that we work with have zero Arab heritage population, so the effect of this is re-coding is to create 50% of our boundaries coded 0, and 25% each coded either 1 or 2. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "FALSE  TRUE \n",
       " 6401     5 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d[ , table(pct_arab > 0.1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Zero Arab</th><th scope=col>1-50th Percentile Arab</th><th scope=col>51-100 Percentile Arab</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.497</td><td>0.251</td><td>0.251</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " Zero Arab & 1-50th Percentile Arab & 51-100 Percentile Arab\\\\\n",
       "\\hline\n",
       "\t 0.497 & 0.251 & 0.251\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Zero Arab | 1-50th Percentile Arab | 51-100 Percentile Arab | \n",
       "|---|\n",
       "| 0.497 | 0.251 | 0.251 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Zero Arab 1-50th Percentile Arab 51-100 Percentile Arab\n",
       "1 0.497     0.251                  0.251                 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d %>% \n",
    "  .[ , median_arab_pop := 0] %>% \n",
    "  .[pct_arab > 0, median_arab_pop := I(pct_arab > median(pct_arab)) + 1] %>% \n",
    "  .[ , .(\"Zero Arab\" = mean(median_arab_pop == 0), \n",
    "         \"1-50th Percentile Arab\" = mean(median_arab_pop == 1), \n",
    "         \"51-100 Percentile Arab\" = mean(median_arab_pop == 2))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>median_arab_pop</th><th scope=col>mean_arab_population</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0      </td><td>     NA</td></tr>\n",
       "\t<tr><td>1      </td><td>0.00127</td></tr>\n",
       "\t<tr><td>2      </td><td>0.00945</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " median\\_arab\\_pop & mean\\_arab\\_population\\\\\n",
       "\\hline\n",
       "\t 0       &      NA\\\\\n",
       "\t 1       & 0.00127\\\\\n",
       "\t 2       & 0.00945\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "median_arab_pop | mean_arab_population | \n",
       "|---|---|---|\n",
       "| 0       |      NA | \n",
       "| 1       | 0.00127 | \n",
       "| 2       | 0.00945 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  median_arab_pop mean_arab_population\n",
       "1 0                    NA             \n",
       "2 1               0.00127             \n",
       "3 2               0.00945             "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d[ , .('mean_arab_population' = mean(pct_arab)), keyby = .(median_arab_pop)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This rescaling into a three-level variable increases the variance that is coded into this contextual varaible. While earlier findings of a lack of moderating effect on discrimination may have been due to a lack of variance in the interactive varaible, this is not the case after this rescaling. As such, *if there were to be a contextual relationship* between the proportion of Arab Americans residing in a geography and the nature of prejudicial behavior by local election officials -- perhaps due to contact, implicit biases, or other mechanisms -- they should be captured in this comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "pct_arab_minority = d[ , felm(GotResponse ~ I(ethnic_cue != 0) * factor(median_arab_pop) | factor(blockID))]\n",
    "pct_arab_felm     = d[ , felm(GotResponse ~ factor(ethnic_cue) * factor(median_arab_pop) | factor(blockID))]\n",
    "pct_arab          = d[ , lm(GotResponse ~ factor(ethnic_cue) * factor(median_arab_pop))] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although there is some (non-experimental) evidence that the districts that *do* have some Arab population are more likely to repond to a request, there is no evidence to suggest that there is a contextual relationship between treatment and this compositional feature. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "==========================================================\n",
      "                                  Dependent variable:     \n",
      "                              ----------------------------\n",
      "                                      GotResponse         \n",
      "                                   (1)            (2)     \n",
      "----------------------------------------------------------\n",
      "Minority Cue                     -0.040**                 \n",
      "                                 (0.021)                  \n",
      "                                                          \n",
      "Latino Cue                                      -0.020    \n",
      "                                                (0.025)   \n",
      "                                                          \n",
      "Black Cue                                        0.004    \n",
      "                                                (0.026)   \n",
      "                                                          \n",
      "Arab Cue                                       -0.106***  \n",
      "                                                (0.025)   \n",
      "                                                          \n",
      "1-50pct Arab                     0.073**        0.073**   \n",
      "                                 (0.032)        (0.032)   \n",
      "                                                          \n",
      "51-100pct Arab                   0.082**        0.081**   \n",
      "                                 (0.034)        (0.034)   \n",
      "                                                          \n",
      "Minority Cue * 1-50pct Arab       -0.026                  \n",
      "                                 (0.035)                  \n",
      "                                                          \n",
      "Minority Cue * 51-100pct Arab     0.005                   \n",
      "                                 (0.036)                  \n",
      "                                                          \n",
      "Latino Cue * 1-50pct Arab                       -0.024    \n",
      "                                                (0.043)   \n",
      "                                                          \n",
      "Black Cue * 1-50pct Arab                        -0.004    \n",
      "                                                (0.043)   \n",
      "                                                          \n",
      "Arab Cue * 1-50pct Arab                         -0.050    \n",
      "                                                (0.043)   \n",
      "                                                          \n",
      "Latino Cue * 51-100pct Arab                     -0.008    \n",
      "                                                (0.044)   \n",
      "                                                          \n",
      "Black Cue * 51-100pct Arab                      -0.0004   \n",
      "                                                (0.044)   \n",
      "                                                          \n",
      "Arab Cue * 51-100pct Arab                        0.026    \n",
      "                                                (0.044)   \n",
      "                                                          \n",
      "----------------------------------------------------------\n",
      "Block FE                           Yes            Yes     \n",
      "Observations                      6,439          6,439    \n",
      "R2                                0.332          0.340    \n",
      "Adjusted R2                       0.107          0.115    \n",
      "==========================================================\n",
      "Note:                          *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "stargazer(pct_arab_minority, pct_arab_felm, type = 'text', \n",
    "          covariate.labels = c(\"Minority Cue\", \"Latino Cue\", \"Black Cue\", \"Arab Cue\", \n",
    "                               \"1-50pct Arab\", \"51-100pct Arab\", \n",
    "                               \"Minority Cue * 1-50pct Arab\", \"Minority Cue * 51-100pct Arab\", \n",
    "                               \"Latino Cue * 1-50pct Arab\", \"Black Cue * 1-50pct Arab\", \"Arab Cue * 1-50pct Arab\", \n",
    "                               \"Latino Cue * 51-100pct Arab\", \"Black Cue * 51-100pct Arab\", \"Arab Cue * 51-100pct Arab\"),\n",
    "          se = list(felm_rses(pct_arab_minority), felm_rses(pct_arab_felm)),\n",
    "          omit.stat = 'ser', \n",
    "          add.lines = list(c(\"Block FE\", \"Yes\", \"Yes\"))\n",
    "         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = capture.output(\n",
    "    stargazer(pct_arab_minority, pct_arab_felm, \n",
    "          type = 'latex', \n",
    "          out = \"./figures-tables/arabPopRescale.tex\", \n",
    "          covariate.labels = c(\"Minority Cue\", \"Latino Cue\", \"Black Cue\", \"Arab Cue\", \n",
    "                               \"1-50pct Arab\", \"51-100pct Arab\", \n",
    "                               \"Minority Cue * 1-50pct Arab\", \"Minority Cue * 51-100pct Arab\", \n",
    "                               \"Latino Cue * 1-50pct Arab\", \"Black Cue * 1-50pct Arab\", \"Arab Cue * 1-50pct Arab\", \n",
    "                               \"Latino Cue * 51-100pct Arab\", \"Black Cue * 51-100pct Arab\", \"Arab Cue * 51-100pct Arab\"),\n",
    "          se = list(felm_rses(pct_arab_minority), felm_rses(pct_arab_felm)),\n",
    "          omit.stat = 'ser',\n",
    "          header = FALSE,\n",
    "          label = 'tab:arabRescale', \n",
    "          font.size = \"footnotesize\", \n",
    "          align = TRUE, \n",
    "          add.lines = list(c(\"Block FE\", \"Yes\", \"Yes\"))    \n",
    "         )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we can look at this more flexibly, using the `inter.binnig` HTE estimates. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p1 <- inter.binning(Y=\"GotResponse\", D=\"ethnic_cue\", X=\"pct_arab\", \n",
    "              data = d[ethnic_cue %in% c(0,3) & pct_arab < 0.02], \n",
    "              na.rm = T, \n",
    "              Xlabel = \"Proportion Arab/Muslim Population\", \n",
    "              Ylabel = \"Marginal Effect\" \n",
    "#                     FE = \"blockID\"\n",
    "             )$graph\n",
    "p2 <- inter.binning(Y=\"GotResponse\", D=\"ethnic_cue\", X=\"median_arab_pop\", \n",
    "              data = d[ethnic_cue %in% c(0,3)], \n",
    "              na.rm = T, \n",
    "              Xlabel = \"Proportion Arab/Muslim Population\", \n",
    "              Ylabel = \"Marginal Effect\" \n",
    "#                     FE = \"blockID\"\n",
    "             )$graph\n",
    "pdf(file = \"./figures-tables/marginal_arab_effect.pdf\", height=5, width=10)\n",
    "grid.arrange(p1, p2, nrow = 1)\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/marginal_arab_effect.pdf' width = 800>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "pct_arab_minority = d[median_arab_pop!=0 , felm(GotResponse ~ I(ethnic_cue != 0) | factor(blockID))]\n",
    "pct_arab_felm     = d[median_arab_pop==2 , felm(GotResponse ~ factor(ethnic_cue) | factor(blockID))]\n",
    "# pct_arab          = d[median_arab_pop!=0 , lm(GotResponse ~ factor(ethnic_cue) * factor(median_arab_pop))] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "   felm(formula = GotResponse ~ factor(ethnic_cue) | factor(blockID)) \n",
       "\n",
       "Residuals:\n",
       "    Min      1Q  Median      3Q     Max \n",
       "-0.7946 -0.0318  0.0000  0.0396  0.7846 \n",
       "\n",
       "Coefficients:\n",
       "                    Estimate Std. Error t value Pr(>|t|)\n",
       "factor(ethnic_cue)1  -0.0280     0.0405   -0.69     0.49\n",
       "factor(ethnic_cue)2   0.0357     0.0410    0.87     0.38\n",
       "factor(ethnic_cue)3  -0.0436     0.0411   -1.06     0.29\n",
       "\n",
       "Residual standard error: 0.46 on 749 degrees of freedom\n",
       "Multiple R-squared(full model): 0.573   Adjusted R-squared: 0.0787 \n",
       "Multiple R-squared(proj model): 0.00549   Adjusted R-squared: -1.15 \n",
       "F-statistic(full model):1.16 on 868 and 749 DF, p-value: 0.0184 \n",
       "F-statistic(proj model): 1.38 on 3 and 749 DF, p-value: 0.248 \n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "summary(pct_arab_felm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.B Political Characteristics \n",
    "<a id='politics'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now turn to examine whether political features shape the responsiveness of local election officials to request for information from our emails. One possiblity for why this might be the case is that local election officials may be politically responsive to the constituents in their geographies. That is, they may behave in a way that is line with the voters in their geographies; rather than, for example, the state law that specifies how the election should be administered. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Political Motivations for Discrimination \n",
    "If it were the case that there were political motivations that were leading to discrimination, then we might expect that local elections officials whose jurisdictions voted more Republican in (a) 2012 or (b) 2016 were more likely to provide less service to minority requesters. Importantly, figuring out this difference involves distinguishing between the *general* increase in responsiveness that exists in the south. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tony McGovern Data \n",
    "This electoral data comes from Tony McGovern, who scraped the data from townhall.com to produce the data. Many thanks to Tony for the work.  There is just a little bit of strangeness has to be dealt with in this data. That strangeness is that the leading zeros in the `combined_fips` field have been dropped. This means that for Alabama, Alaska, Arizon,a Arkansas, California, Colorado and Connecticut we're not matching FIPS. No matter, we'll just correct that. \n",
    "\n",
    "We have one county that is not matching. The Oglala Lakota county in SD. The rest seem to be matching. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "results2016 <- fread('./data/County_Level_Election_Results_12-16/2016_US_County_Level_Presidential_Results.csv', \n",
    "                     colClasses=c(combined_fips=\"character\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the `clinton_voteshare` and `trump_voteshare` two-party vote-share indicators. These are just \n",
    "\\begin{align}\n",
    "clinton\\_voteshare &= \\frac{votes\\_dem}{votes\\_gop + votes\\_dem} \\\\ \n",
    "trump\\_voteshare &= \\frac{votes\\_gop}{votes\\_gop + votes\\_dem}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "results2016 = results2016[ , ':='(clinton_voteshare = votes_dem / (votes_gop + votes_dem),\n",
    "                    trump_voteshare   = votes_gop / (votes_gop + votes_dem) )]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Romney Vote Share \n",
    "Here, we create a variable called `romney_voteshare` that is the percent of the voteshare won by the Republican candidate in the 2012 Presidential election. It is a simple transformation of the `obama_voteshare` variable that we used to block. We make the transformation only for comparability to the `trump_voteshare` variable that we will report in the next section. \n",
    "\\begin{equation} \n",
    "romney\\_voteshare = .5 - \\frac{obama\\_margin}{2}\n",
    "\\end{equation} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHgCAYAAAB91L6VAAAEGWlDQ1BrQ0dDb2xvclNwYWNl\nR2VuZXJpY1JHQgAAOI2NVV1oHFUUPrtzZyMkzlNsNIV0qD8NJQ2TVjShtLp/3d02bpZJNtoi\n6GT27s6Yyc44M7v9oU9FUHwx6psUxL+3gCAo9Q/bPrQvlQol2tQgKD60+INQ6Ium65k7M5lp\nurHeZe58853vnnvuuWfvBei5qliWkRQBFpquLRcy4nOHj4g9K5CEh6AXBqFXUR0rXalMAjZP\nC3e1W99Dwntf2dXd/p+tt0YdFSBxH2Kz5qgLiI8B8KdVy3YBevqRHz/qWh72Yui3MUDEL3q4\n4WPXw3M+fo1pZuQs4tOIBVVTaoiXEI/MxfhGDPsxsNZfoE1q66ro5aJim3XdoLFw72H+n23B\naIXzbcOnz5mfPoTvYVz7KzUl5+FRxEuqkp9G/Ajia219thzg25abkRE/BpDc3pqvphHvRFys\n2weqvp+krbWKIX7nhDbzLOItiM8358pTwdirqpPFnMF2xLc1WvLyOwTAibpbmvHHcvttU57y\n5+XqNZrLe3lE/Pq8eUj2fXKfOe3pfOjzhJYtB/yll5SDFcSDiH+hRkH25+L+sdxKEAMZahrl\nSX8ukqMOWy/jXW2m6M9LDBc31B9LFuv6gVKg/0Szi3KAr1kGq1GMjU/aLbnq6/lRxc4XfJ98\nhTargX++DbMJBSiYMIe9Ck1YAxFkKEAG3xbYaKmDDgYyFK0UGYpfoWYXG+fAPPI6tJnNwb7C\nlP7IyF+D+bjOtCpkhz6CFrIa/I6sFtNl8auFXGMTP34sNwI/JhkgEtmDz14ySfaRcTIBInmK\nPE32kxyyE2Tv+thKbEVePDfW/byMM1Kmm0XdObS7oGD/MypMXFPXrCwOtoYjyyn7BV29/MZf\nsVzpLDdRtuIZnbpXzvlf+ev8MvYr/Gqk4H/kV/G3csdazLuyTMPsbFhzd1UabQbjFvDRmcWJ\nxR3zcfHkVw9GfpbJmeev9F08WW8uDkaslwX6avlWGU6NRKz0g/SHtCy9J30o/ca9zX3Kfc19\nzn3BXQKRO8ud477hLnAfc1/G9mrzGlrfexZ5GLdn6ZZrrEohI2wVHhZywjbhUWEy8icMCGNC\nUdiBlq3r+xafL549HQ5jH+an+1y+LlYBifuxAvRN/lVVVOlwlCkdVm9NOL5BE4wkQ2SMlDZU\n97hX86EilU/lUmkQUztTE6mx1EEPh7OmdqBtAvv8HdWpbrJS6tJj3n0CWdM6busNzRV3S9KT\nYhqvNiqWmuroiKgYhshMjmhTh9ptWhsF7970j/SbMrsPE1suR5z7DMC+P/Hs+y7ijrQAlhyA\ngccjbhjPygfeBTjzhNqy28EdkUh8C+DU9+z2v/oyeH791OncxHOs5y2AtTc7nb/f73TWPkD/\nqwBnjX8BoJ98VQNcC+8AAEAASURBVHgB7L0J3BxVlf6ffQMEDPsaksgiIKOI6LgxKijuI8Zt\nRNkM2VQYBWdRZlwHHZH5+8tuAkEURNxmXF5FkCAYdUYcBAIIkrAKIYLIQNY3yf953vTpnL7v\nreqq7q56e3nu53Pee8+5555761vdfd6qrqoeNkxFBERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\nBERABERABERABERABERABERABLqawPAh3LqxmPtoyDGQ50B2gkyAfBayCsJyCGT1QEt/REAE\nREAEREAEmiLAhLsI8hfItoi8CDYrt6HxM8hLzaBaBERABERABEQgH4ExcP8cpB8SS7xmswQ8\nCn6bnO/fo60iAiIgAiIgAiKQk8Bc+FuSTastAR8R8f9QzjnlLgIiIAIiIAI9TeDN2Pq0pOv7\nLAG/PTJmC2z8vlhFBERABERABDqawIgSVr8f5rgkYZ6tsD+c0HdkxM71fiRil0kEREAEREAE\nRCAgwKua/REu23dD3gQZB+ERre+3I+CpsM+DMEn7/vXQd4GoiIAIiIAIiIAIpBD4Ifp8Av09\ndN5yZCUpAVv/2Wj48Wy/wDpVi4AIiIAIiIAIxAk8BLNPoG8I3Ool4PHwfzqI8bYghlQREAER\nEAER6CgCRX8HvAdo7B8QWRHo9VSecr4hcDow0KWKgAiIgAiIQEcRKDoB7x6h4U8/R7qjpn0D\n67pAlyoCIiACIiACHUWg6AR8L2g8ExB5VaDXU/k4yqMCp8cCXaoIiIAIiIAIiEBA4Cbo/jvg\nP0E/2PmkfQfMZ0PzlLUfz/ahbryaIiACIiACItBxBIo+AiaQ/w2oTIR+K+RjEP4Yw86QsPB7\n47+D3Al5SdB5D3TexqQiAiIgAiIgAiKQQuAw9PE0dHgUm6TzoqukPto/AVERAREQAREQARHI\nQOAs+KQl1ax9qxCHtyWpiIAIiIAIiIAIZCTwLfhlTbQxvycx/sUZ55KbCIiACIiACIhAhQAf\nO3kBJM/paEvEPPI9thJHlQiIgAiIgAiIQAME+BCNb0AsuabVvAjrfZBREBUREAEREAER6BoC\nw4dwS/bB3LxAi7cU8VakqRAeHfNodzXkDxDegrQVoiICIiACIiACIpCTQBm3OuVcktxFQARE\nQAREYGgJjCxh+jmY41LIbpAHIbyYSkUEREAEREAERKBgAnxohn3Py9PJfDJW3sdRFrxEhRcB\nERABERCB7iLAo15LvlYzCfMJWCoiIAIiIAIiIAIFETgCcS3xWn1HQXMprAiIgAiIgAh0DIGi\nL5Di6ec/BzQ2B7pUERABERABERCBAgjE7vmdXMA8CikCIiACIiACIuAIHIQ2r362U9Csfwzh\nryKpiIAIiIAIiEBPEijrQRwHgO7lkBMcZf7q0fcgfNjGAxD+TvDTkA2QLRVBFS33Ra0yioAI\niIAIiECHECgjAf87WDwfwu+bXwFpxb3HZawbS1URAREQAREQgWIIlPGM5Rdg6a8qZvnNR50z\nZ85+W7duPd8iob1k4cKFt5veTD19+vSDRo8efezw4cOPQZxHEfvm1atX39rX17exmbgaKwIi\nIAIi0PkEykjA7U7pUyNGjDjTFon2z9BuKgEj8U4YM2bMEsR5t8VljdjDpkyZsmbmzJnTFixY\ncKPvU1sEREAERKC3CPC0cM8WJMKTt23bdlorAcyaNWsfHPX+CjFrkq+bY28k4uvgV036rk9N\nERABERCBHiHQswmYyXfkyJHfxenhVnwnXX25IN5FkPBJX09VHdBA/2gk4Xk4Up7i7WqLgAiI\ngAj0DoEyTkH/G3Be2i5ITz755LGTJ0/+JJLgeVhTS/8BQUI9HHHf47cVR9izVq5c+ZWpU6ce\nOHbs2KvQf1ylfyyOlD+O9uneX20REAEREIHeIFBGAr62HVAiOSLfjT4LCfAjWE8hR56I/w6/\nrUi+K+bNm7eAtuXLl6+ePXv2R9G8wXywlrdOmzbt7KuvvnqT2VSLgAiIgAj0BoGWHgG2MzKc\n8t0fCW8+1uiT7xroa1u1bsQ/KYh1hdeRjG+E7k9H7zZx4kQ7IvauaouACIiACHQ5gZ5JwJH9\n+DPcFvRXOEp9ONLXqOnIYCCfhe0LnwJ2jzeMGjUqHOO71RYBERABEehSAp2UgHfGPjgVwsdY\nNlyQcFch8c6eO3fuifPnz3+04UDBwBkzZuwFE39+sVow10NVpdIIbdAPC32ki4AIiIAIdD+B\nMr4D9hQPh/I6yCEQJqtdISMhfPQkfyfYhDqfdjUWMh6yP2QqZBykobJ58+ancLT5dpwG/i4C\ncJ6WFpx+3j0MuGnTJj5Ws6bAL7TVJO0aZykiIAIiIAJdS6CsBLwTCC6DvH2oSC5duvQJzP3t\nAufnEXpNwcM4+LzrmoIj3g1Iwt42aJzvTGg/G/YLIWMS+kMz/SZDXhx2SBcBERABERgaAmWd\ngr4EmzdkybcMtEiqPFqvKbAN+u1j2PprnJo4qg/ipKk803A8JGvCToulPhEQAREQgRYQKOMI\n+ESss+b2nBasu+1C4Hvl9XzUpC847R1LeGGiXufHZGzzaH56Rl+6vQTy+hz+chUBERABESiY\nQBkJ+F0t3AZ+f3pNC+O1LBSeqvVMGAwJOUy2w3AKelxwCvrpcJx0ERABERCB7idQRgLmRVdh\neRyGWyH7QI5wnT9Cmxdg8cKrSZCpECt/QeMwCO/dbcfCbaopSLYTagzbldDWsvuQI3PJJAIi\nIAIi0KYEas+Ztn6RTDb7BWF/DP1gyKsg5wZ9/wH9zRCetn4OhKdZ7RQtv8e8ANKWBVdXMwHz\n1HC14KrrSVVlR4NXgPtyp1fUFgEREAER6A0CRSfgQbfmAOunIXa69nq0/8+hZlL25StQ/BH0\nTOhMzm1ZcMR7h18YdH90P4yPw8TpZ/8kLp6Svs2PUVsEREAERKA3CBR9CjqWgO9yaPkM5F9B\nLKm+1PVZk49v/AnktRDev/NFyDGQ0gsS6L44qq0yW7du3dply5bxe2krfWi8zBQk22loX2Q6\nhn4AbX9h1kN4GMjvrF+1CIiACIhA7xCoJpOCNpnf54bFJyz2/RpiCZjPRR4HCX1+ARsTMMtR\nEN5XbEfRtJVS8GML1yOp8nvogTJ+/Hiuu/pjE/39/VcjyX4KPny4CH928Hj8AMPFuEJ6IdpH\n4qKsmlPoOPq9Em58PKWKCIiACIhAcQTsc7bmIQzFTZctctGnoP8YWQZvifGFCdgKk2/Yzz6f\nyLnmv6Kx3cqiRYvuwZoW+3Uh8Z6DK6TvQvL9Nux7Wx+S7+MbNmy40HTVIiACIiACDRNggrUk\n23CQsgcWnYD/gg2yi6hs28L7UX+JDg/uveZYqfkfy5sC25Ccgg7WEFVXrVp1LpLrsmjnDuNa\nHBWfUnk61w6rWiIgAiIgAkUQYB5pq6NfbmTRCZhzrOIfV85F+7MQ+36YVw/774XfD51XP4+u\n+PCI8sUQX2quNvYdQ93u6+vbiCuiT8c6zkEifjhYTz9sy5F8j12wYMENQZ9UERABERCBxgi0\nZYKttyll/EfAH6H/98hC/hW2T1bs81HzCmdfNkIZ6w2u/QK0/9fpbducM2fOfky4kDU45Xxr\ncNFWWevmaf0VEPLkhW8qIiACIlA0ATuz2Uie4dhGxhW9TR0Xn0e6vGCKQL38rdsSXjns+9La\nvG0pfJiFC6VmhAATMJn6K7AjbjKJgAiIQMME+BmTp6T5+xyQJ6Z8IwTeD1s/xEOtuR8WfT8P\n+r2vb58PP5V8BJSA8/GStwiIwNAS4Gc+i332b9e269ZWnYMA74ldDyFQHsWGpxd4pLyy0m/Q\nw/oW9PO7YZV8BJSA8/GStwiIQPEE7PM9nMnbfZt+1FUaJLAnxn0C8p2E8QfCzkczGnRfL4Rd\np54TwNUxKwHXAaRuERCBlhOwz++kwM32J8XtGHt4FNoOC+eaToQ8H8LnP98O+Q3kbohKYwSY\ngHURVmPsNEoERKAxAkywLFnzjPe3dp7xA5Ppjwi0GwEdAbfbHtF6RKB7CNQ7kvVbmubr+3zb\nj++qdtGPouwqWNoYERABERCBAQJMkCxZj27pa2PYrlfyxK0Xq237y3gQR9tuvBYmAiIgAiLQ\nNAEmy6wJM4uvJerwKNjsTS+4XQLoCLhd9oTWIQIiIAKdQyBrwrUt8snTty1OzGZju7YuOwHz\nQRCnQF4IOQAyHsIfYBj49SDUWcurszrKTwREQAREoBQCTKJMqFZnmdQSr42zMWan7uP5tvmG\nPt6udoXAeagfgRBgs1IJqSojAV2ElRGU3ERABDIRsM9w72w21r54u7XZ79vmb2OtL6zNT3UO\nArPgG4JsRs8xtVxBQAlYLwMREIFmCfAzm8U+u7drtX+tz3xDf+s3e+3oHbHNz8cJfaVnIMBf\nMtoI8UCbbWeYVi6OgBKwg6GmCIhAXQJ5Ep99njOotevV4QLMPxYj9O0avYzvgPkrR/oRgK55\nyWhDREAEeowAk6MvduGUt7FtfuH3ueyL2fwYtsOSNE/o17F6GQn4uAQ662C/D8KaR8j8sQYV\nERABERCBoSeQlDBjK7NEaQnY6phvaIvNY/FC3yTd5ss7Lile19h3wZZsgRCQCX+Q4QyIYAFC\nSUWnoEsCrWlEoIMI2GeyJbBw6b4/1qa/2X07yRbGz6P7mOG4tL7Qt6f052BrDY7VH+opAu2x\nsUrA7bEftAoRGEoC9hkcrsHbfZt+STrtLNafpd4+YscY07PUFj/Jt15/0rghtRf9JKw/YOue\nDLbwmkCXKgIiIAIiECfAxNKqwrOO/sxjWtKyef0Ys9l6qPt+syfZrL+ROhbTx6nX733bpl10\nAuYO+lWwtX8OdKkiIAIiIALZCYSJMPvI7Z4cb8nTj/U2JjRfrC9mZ58v1EOb7+/IZOk3oFXt\nohMw13lDsNjXBLpUERABERCBOIEw4dErZouPTj/d65OkxTSbJVGrGd+3k+YL7YxLaWRsGEt6\nAwT4qEn+nq/tgO+jnffRkw1MqyGOgL4DdjDUFIEeIWCfuazD4vusbT6m56k5Nsnf99kcVnOM\nFRtvuuoWETgQcR6DGOCfor1Hi2IrTH0CSsD1GclDBIaKgE9CWddgn6Vp/jEfs8VqixXry2ML\n45geqxnXis1hetfXrbgP+HhQmpSB1NXw4SMpWXgaeiXkWsjvILxQ6/8gWyFZylVZnOQjAiIg\nAh1AwE7/ZlmqT1hJ/t4nT+ykeD6Gjx3z976+HfOlzfv4dpK/7AGBy6Dbfy5l1cESpNYhoCPg\nOoDULQIdQMB/vsaW6/uzthkn9A1t9XSON59Kc1BlcwzqcIYsPs6985utOALufAraAhEQARFo\nXwKW4Mo6QrT5jEiom93X5mN1uNZQ92OT2kmxkvw7zt6zCXj69OkHjR49+tjhw4cfg7326Nat\nW29evXr1rX19fRtbsRdPPvnksZMmTXox4k+FTEbMx7dt23bPpk2b/nfJkiUPtWIOxRABEegJ\nAo0kL46xBEZIoW7gvI/ZstZ+XbE4ZvN+abGz+qXF6Ki+VmzwZdji95W81Q2vG4l3wpgxY5Zg\nve+OrHnNli1bpi1YsODGSF9m06xZs942YsSIL2LAIeEgJOHNsH35ySef/NTXv/71p8L+gnSe\ngl4BGQvZVNAcCisCIlAeAUtuNiM/E70tTU/rs3i+TvKn3Uo4t9lVpxAo4z7glOnL7UJi3AdH\nvXwwSCz5cjF7I3FeB78zG10Zxs5CjG9j/CGxGDgaHg35yG677XYt/hkYHfORTQREoOsJ+ITV\nyMb65OfbjEU9jO91306a28ekP3UTjvH91FUaINCKU9DnY97PNDB36UOQ+C6CHB1MzKPQZ5mt\nkiDnITkuX7x48b1mz1LPnj37UIznkW9NwVHvrTAcgj7+OMVAQfs4/DNwAZRPVEyqREAEeouA\nJcJ6ycwSoNGxcV73McJ+80urbbyNtZp2a2cZn+ajvoBAKxLwGsSktHVBQj0cSe89fpFIjLNW\nrlz5lalTpx44duzYq5gUK/1jkRw/jvbp3j9D+y3wGW9+iP97tE+ZN2/eSp76HjVq1D/h6Pif\nrR8116ME7ICoKQI9QiCW2CzR+T62WXzfdkvtX+uvtcY1H58eHGvzxEak9cX8Q5utrdk4YdyO\n13vmFDQS6jv83kJyXIHEuGD58uX9uChqNfo+6vuRjN86bdq0Md5Wr40xz/c+mGMRky9tOJrm\n7x5fCHmGeqUcgsS8qymqRUAEeooAE1KepNSMb72xliTDHZBkN796/fTLu50Wu+vrnknASI4n\nBXvzCq8jUfLCK39R1G4TJ060I2LvmthGwvXjh2FOnnqulvnz5z8N5f6qYdiw9bgqeqPT1RQB\nEWhfAlmSTTOrtyTJeZKSVtY1hH5e9+3YesO56/n7GPT14vvUDgi0awI+GOt8FWTvYL3NqEcG\ng+8OdL5o7vE2nDIOx/juQW3cyvTjwFhz1D1jxoyj0P9c5/OzZcuWbXC6miIgAiJgBCyRUWdS\npN7qYkmfcfPEN18/vtVr6/p4rfgOOC+kV2IAHz3JR1CG5TQYLoI823WsQnsu5GJny9VE4tsL\nA3bzg3C0+pDX2aYNR63Hmh36YdbOUm/cuPG68ePH/wkxBp5zjfpUXJi1Ebc2XTVy5Mi/QoyZ\nPg7i1xyF+z61RUAE2o5AWrLJk5DMlxuYFNP70I+69w376cNiPmE/7aGN/mZL6qdPnmLz5xnT\ns75lHgE/D5R/DVkOeSMkLEy8l0J88qXPZMiXIN+A7ATJXZAIdw8H4dTvoCNP+IW2mqQdxgj1\nSy65hM+zfiMS6x8rfeMR84M4kr4J9VyIHVGvh8+7cUr6yjCGdBEQgY4mYAkt60bQ30vaOPNr\nxCdpXUyYYdJM8vXzNjLGj1cbBMo6Auap1+sgA0eGqPn0KV+on+MNkfY7YeNDLE6N9NUz7Rw6\n4GEc60MbkuIGJElvHjTOd8ba+C7512efffbrkXRvRKzqbUeB76W4+vpbgS2Pyn9SeEHXmIyD\n9szoJzcREIHGCPCDo17isv4svq3ysa1Jisc1sY/Far/O7T36WwiBMo6A+bCJPoglX24IT8f6\n8gUoWdbC23bsKNKPT20jEfIJUDUFNibzmgJbf41h2LBxgV5XxSnns3DF9a9Tki9jzDryyCN/\nyweD1A0oBxEQgU4hwARmSYxrZiKzZEbdSsxmfVYn+YTxzb9eHcbzccKxaX2hr+mNjLGxPVuX\ncQT8JtA9ICA8BTpPJ/OWnP0hJ0GyFCbpT0PelsXZfHBx1Hrcf2vqQL158+bY0WOYqHnrUOYy\nc+bM18B5IWQkB+GIejMS8Sf7+/uXo94TazgfNR8LySukj0Z1DW5DOha3KA36Z4A+KeUJ9E1P\n6Q+7OOfrQ6N0ERCBlhGwBBdLRNZnk9GHNqvNbnWSnf1hLBuTpba4VsfG+Pj0UymQQBkJ+AOR\n9T8NG4+ImYDfGOmn6bOQ2yD/APFHzG+BzuS5CZKp4AIozlNTkAzDZMuEOQ6J0ftxnZkLYn4K\n4weSb2XQ+XPnzv0PC3DCCSf8gEe+leQ7kIRxtMyj+svMR7UIiEBHEPCJiguu+eDIuAU+Bseb\nbnXGMLncwnV6PWle2r1frgnlnEyg9rAw2a+ZnuOCwT+HzqO/+yv2WAL+Ffr4JKqrIH8D8QmU\naz4Ikqc8Hjoj2U4IbdBD29qIT9Q0Z86c/ZBYB45uKw5bcZTN9VcLH/qBef9f1bC9wTMEKiIg\nAu1HgInHkpJvN7tSi+njxGzsZ+LLm/yS/JPmsHXYXFa3cpttjnr1UMxZb02F9RedgPn9b3hV\n8/mwPVDZIh6FvrrS9tUPnfIk2j9xOpuTAj1VxYVRTMA8bVstuEhqUlXZ0ThkR3OgdWegJ6o4\nzVwzFon2TpxafiQcgCR9X2A7ONClioAItDeBMJExYYUl9An7s+qWkGJzxGKYH2sv5mvrMj/T\nrT9Wm2+sz9YX62vEZmtuZGzHjSk6Ae8NIn7n8Uj2ZkfpRWiPd7o1f26NSm0J28yTrJG1RkK8\nw/tCP8Lr+C6Wv1I0xdvgw1PgmQrG/ilwjF5ghZj7Bn5/DHSpIiAC7UGgmWTgP/dasTVZEiXn\nsYRodTg315XUF/Nt9XaEc/S0XvR3wBMDunz4hr/S+BVBP1V+t/vfgd2PYVfuq5Mxhldiv4yD\nWZAwp6G6aEDBHxwR87tqfrds5SHcp1vzsBAk6X3hV2W2bt26tfYkK/yG8N04Df0UBj+LARB/\nIq6Ifj2Ovn9kAVnje+J3eD1Pkvfj1BYBESidgCUjS2K2AJ8czcf6stQ2xsfJMi6rj8UN123z\nZo0T82tFjFjcnrBVk0lBW/tYEHfnQD8x0Kn+GrIhsB8c6A8Fel0Vp4ivRu6sXiSFBHk8EuTF\nuEJ6IdpHIjHypwGrBYnxSij2wh2w44Kp6+F7mDnhqVdc/7UVHUO2/RL9r7V+tC/BldFn4klY\n1+BCsJ0hvKDsDdbPGvPaeG9WWwREoFwCfK9bMrH3fZiwYisKfWxszDfJ5seE8ZLG1LPbtni/\ncB7fF2ubfyxWzF+2nASKPgX9KNbjr1Y+GProyhp5dPzSSttX4fe9/A75ZO+A9oOBXlddtGjR\nPXBa7B2RIM9BUrwLSfDbsPN0+UBBIn18w4YNF5qetcajKGfC13/XvDfi/wCJ+xnUtJ/vY2Ge\nS3GV9HJvU1sERKBtCDABJSUfs5uP6VkXT//YGEt6rYwTmydrfPkVSKDoBMwXkz9a3RX6LAjn\n/RgkdgTOU8VWno/GDyDhkXP4nbD5p9arVq06F0lvWarTsGFrcVR8ytKlS30irTNkezd/1hDx\n3wXN/9PB09H2T4eP89+4Svpcb1BbBERgSAnw88oSoE+QsQRmfrbgUDd7Uh3OFfrZ/LG5Q98k\n3eZgHcbzfUnjbUxSv+xNEig6AXN54YVMvC+Wp6bPY2dQ/gD9txUbf6j+NxB/aw+7eItS5tuD\nOMBKX1/fRnwnezr0c5AoHzZ7peYtQsuRfI/F97k3BH2ZVcT/KeIcClkG2RIOhO3PsH0IR74v\nwVXSfwn7pYuACJRKwBKRJRuruQjrK3pBnCdWssyf5GPbYbWPbzarfZ/aJRLgDii6MIGuyDgJ\nv4f9dMV3KeozIuPeCds3I/bcJt67y4QLWYNTzrfaBVW5AyUMwEVbE3DqeSqOgKfiNPcz+AEI\n3prE0+dJb7iESE2bbR+MRaSao/OmIyuACHQmgdh7kJ+HMTu3MPys9H5+nPn5/hghPybszxIj\nHB/qfs3hWix+OK/0LiXA08p8EaQJ79XdzW0/vy8N/VfDFjtt7YapGSHABEyW/irviJtMItAz\nBOyzxW+w2bLWWcaaT9aYWf3CuNRtrPVZbXarza66RwhMwnbeArEXQFivQR+/7/WFj5z0fjxt\nfaR3UDszASXgzKjk2MEE7PMi3ARvj7XNFtY+Ttjndfp53do23uvWbkXt57W50mqbM81HfV1K\nYDy2i1ch88lW9kJYj/b1kOdAwnI4DObHq6mPDh2kZyagBJwZlRw7mIB9XoSbENpNb2Vtc1rM\nUPd2a+etGTM2xuZSLQJ1CfD7BybX4yFpp0R55fBdkDmQnSAqjRNQAm6cnUZ2D4EweXHLQlua\n7kl4vzQ7/az4MbG2+Vkd+pjdauun7tvWr7rNCQzF96l8oTCx1iv8ib4jIPRXEQER6G4C9j5v\n5QVCSTHtgiXr92Stz9usTf9wfaFuvqwtvtW+L6md5hvOn7bWpPiytxGBMm5DamZz016MzcTV\nWBEQge4lwM+N8LPDkldSwqTd+sLakwpj2zxWe1+L421hO4sPx6T52dxpPuG80tuAwFAcAdtm\n85YYfq97DITfAfM08wTIZyGrICyHQFYPtPRHBESgmwm0MnkwFpOS1ZagyM/a4Xzen35hP21J\nxcYm9afZbT1JPrYN7I+tyfcnxZBdBKoEmHAXQfgQCr74QnkRbFb4EI+fQWKPrDQf1fUJ6Dvg\n+ozk0ZkE7PMjafXWz5rF9O3aDt36vY/ZbEzeOjaHxQjnSZrLYoS1jxP2SReBQQR4wdXnIP0Q\ne/HEakvAPDrnQyPM5+/RVmmMgBJwY9w0qj0J8DPBin0+UDe72Uy3Pm+3PrOF8azf2y2O1Ta2\n2drmsNrHM1un1Lb2Tllvz6xzLrbUdk5abQnYLsDyvh/qGVqt3VAl4NbyVLShJWCfCX4VZrPa\n91mbfSzmU1adNmfYR53F1ha2BzrdH+9nZtp8CXXf1+p2bD2tnkPxchJ4M/xtx9SrLQG/PTKG\nz1aO3TOcczk9564E3HO7vOc2OPa5Qgi0++L9aPd6q9o2X1q8PD7mm7XmvL6Euu9TewgJlHEV\n9H7YvksStnEr7A8n9MWeesX1fiTBX2YREIHeI2BJLnaBkiUe8zHdKHmd42MxzDdrbRdF+dh+\nrM1ja4r1eVtSHO8TtsPtCPXQX3oXE+BVzfZis/pu2N4EGQfhEa3ZWdsR8FS050GYpH0/n561\nC0QlOwEdAWdnJc/OIuA/G5La3KKkvjx2I5NnTD1fixnWHMdi9XZNf0UgJ4Efwt+/CH8P3T/Z\nKikB2zRnB+MZ6wXWqToTASXgTJjk1IYE/GdHmIzCviSdmxX2hTavs82SNibWH/pn0Qcmqvwx\nf4vt+8po+/nLmK/n5yjjFDRvO/KFVzM/4w112l+N+E+qM0bdIiACnU+ACSEsSUnCTu2G/jE9\n6ylZ78e2X49v55kjjOnHss/6rfb9Se0kJkn+srcJgaIT8B7Yzv2DbV0R6PVUnnK+IXA6MNCl\nioAIdB8BS0ix2m8t++slIR/D+9LO4hOqb2/v3dFv/mZPqmMxYr5+LWF/1hjhuEZ1Y9ToeI3L\nSaDoBLx7ZD3+9HOkO2raN7CuC3SpIiAC3U0glqgaSRg+qXG8141g1iRLf+9rbautn7oJ57P+\n2NwcY8X8TE+qLXZSv+xtSqDoBHwvtjs83fyqnCwOgf9RwRj+NrCKCIhA7xFg0vJiBCxZ1UtG\n5mfj6tUxf2/zSdS30+LSz9ZpdZr/UPcZ76FeR9fNX3QC5hXMtwTUvgT94MCWpPLZ0F+HjA4c\n7gx0qSIgAt1NIExUoc6tp43FJzjTLYlY34Aj/tgY031tY+rZfD/bNofFpp5WbB6rY75pfTF/\n2TqAQNEJmAj+N+AwEfqtkI9B+GMMO0PCwu+N/w7CRMsreH25BwpvY1IRARHoPQKWeC0h+Zo0\nqFvxbbOxpj0tOSaN41gbx3ZaCWOYbuvnWLNZHN9ntlbVnCucL2vsIteVdQ3ya5DAYRjH09D2\nAqhX86KrNJ9PNLiOXh6m25B6ee9357bHPiO4pbRbifmYLeZDm++3dqz2vuwP9Zit4ladI/Sx\n/iJq24YiYitmmxM4C+uzF0Az9SrEGd/m29qOy1MCbse9ojXVI2AJKubnP0d8v7eHbfqFNuq+\nmB7z8zaO8bpvWzyzmW51XruNUy0CDRP4FkbaC6+R+kmMf3HDs/f2QCXg3t7/nbL19rlg66Xu\ni/WHNX1CWzjWx0nyj8WI2SxWrM/bzC9rbWOz+stPBDIT4GMnL4DkOR1tL0ge+R6beSY5hgSU\ngEMi0osgwPdrM8Xe71aHscyet2YcjvGlkRh+PNv1YoT+0kVgyAnwIRrfgNR78bKfF2G9D8Lf\nBlZpnIAScOPsNLJYAvY5YLOYzjpWfH/Ypn9o83q9fu+b1LY1xfqtT7UIZCKQ9Yq+TMFyOu0D\nf16gdSiEz4Pmjy/w6JhHu6shf4DwqVlbIS0v06dPP2j06NHHDh8+nI/KfHTr1q03r169+ta+\nvr6NrZxs9uzZB2/btu24ESNGPA9x16J918qVK69fvnx5fyvnqROLCZgsx0I21fFVtwiUSYCJ\njMU+i0zfbt3+l31mT/IzO0eY7/bRrf3r54nNZf22BtPDVdTrD/2ldyGBpBdHF27q9k1C4p0w\nZsyYJdDeHdnINVu2bJm2YMGCGyN9uUwzZsx4/qhRo67CIP5zEZa7kPA/PH/+/GvCjoJ0JeCC\nwCpsKgEmGf8Z45OOb4dBrC+0U2c86/dt62Nt/WyHJRzj+30f22GpF9f7m28sDv3q9ftYancp\ngTLuA24bdLNmzdoHR72/woJiyZfr3BtHqtfB78xmFo3xp40cOZJHnLHky9CHY54fwu9lzcyj\nsSLQwwSSElsSEvqnjWGfJUWLQd2L2X1dL6739e1Gx/kYbNv6Qrv0DiBQRgJ+HTic4eQVObnw\n6O3jkIWQH0AugTRUcLr5Iggf/uHLU15B/2gkx3k4Up7i7VnbSKpvx/hLEYcXnQ0UnHbegsaa\nimrVKPh9c9q0abqtyoio7jYCYcLzScfalkBYW4mNsz7W7Ldx3tds3tfa1ufnsT7WoT3UvS/b\ntn5vtzm8LdbO6hcbK1sXESgjAZ8HXkud8J7gPIXfnX4awt8FfgPkVEjsRx5gTi5IqIcjKb7H\neyAxzrr99tsnbtiwYTLa/+P6xuJImUk/VznttNPGIal+2Q9C3P/avHnzxLlz5+7T39//Muh/\ncf377rXXXic5XU0R6HYCPvmkJTmfWMnEdD/es7J+s1mC9LX1NVuHc1k8m8u2K8nP/JO2xfqz\n1DZnFl/5tBmBMhJws5t8UxCAV0QfH9jqqkio7/BOSIQr5s2bt4AXQy1ZsmQ1+j7q+5Gs34qj\n0zHeVq+98847vwk++5of5vjzY4899q7FixcPJN2FCxf+An2Xuf4/oR3+XrJ1qxaBTiVgCShc\nv7f7tiUR2ryd45OSmI3xc8Rsvt/aWf3MP0vt114vvvVbnSW+fLqQAJNZu5fJkQVWk1ykL2pC\nQg2PNK/wjkjGN86ZM4eno59Vse82ceLE49Bm0sxUkHDfjnmqvtC/evXVV/PRmtXyzDPP/Asu\nAvvypk2bHl62bNmGaocaItB5BCxZ7njRb9+GUE/bMvqGcUznOGv7mH5MUmyO82PoZ+MsZjjW\n+mN2s3Gsj50Uy/xVi0AigVYlYP7C0R4Jszw3sL8U+lcDW6jyjcC18YcaXh12QrckGelKNB0Z\n9IQ/6MA3En/oofrAD1zFzDGZEzCSb3Us54LOC75qCpIun+hFURGBXiBgCYrvaYoVn8TM5mvz\ntfG+L9aO+Xkb43k9LUZs7npjGc/GxWLLJgKDCLQqATOZnTsoetzAI9rYUW3cO279fdwct+KW\noL3Qs5vvxdHpQ15nmzafRKHzPuVMhbc3wX+yPwLGwNW4KOskfC98GtovgjwNH37X/A0ccV+H\nWkUE2pEAk02WZBLz8WPDpOX7YmPJwsZ4X2/nOPOhPVYstvezeN5mY8OY5mv9vrbYZgt1s6sW\ngboERtT1yOawGG6/y+baEq9b8kRBUtw99Mcp4EGnf+EX2mqSdhjD6/iOeU+Mr3kzQn0v5Mfw\nezeEV1UfA/0s1D/FAzq+iKQ9Gm0VEegWArHkxm2reV8EG2tj6OPboW7DLDmy3+Ja7X2s7WuL\n721sm93HNJv5+j6zZa0ZK4yXdaz8uphAq46A+bSqD0OWl8DqGszxaM55Bv3mML6HrflulvFw\ndLohyKGDxiXNiwd47IJT1mH3nCDeQD9sfDN/BEmbF3l9KByUQX82fC6EcHyWsmcWJ/mIQIVA\nmNAMjCWRpH7zs9r72VjrYx3azD+0x3TzZc1+0xnXitnC8dYf1hbHYrLfYoS+Xrf4oW+S3Y9V\nu4cJtOoImAhvgHyzYJaPIP778s6BfMdHMNYU2DbXGKDA1h/YqvfyBvaYOiFmRFL/JeSVOOI+\nAvUnIFuc3ywcCb/A6WqKQDsTYILxSYYJxkva2m2cJaWYr8UK+8J5rT/JP2u/xbW1cZzFtD6L\n1UjdihiNzKsxHUJg0CFbk+vmPb9MKP7U6t7QfSJ7Bjpvv0krfBMwUTFJPg25H/IdyPcg6yC5\nCh77uB7fw9aMwb25saPHMFFnngtHv4OOqDHhmqeeeurEyy+/nNvM8hl8JzwOif6fqaAeiYrt\nU6jnKE/Ad3oO/5fA9/U5/OUqAo0Q8IksNj5vPz8HrLBdb7z35zj6e1tsvPVbn+lWWxzWZjNf\n2lhCfbtVf0WgDoFWJ+AHMN9zgjl5sdGrnI2JNPdRrBufu4nHQloCrI5FQg6TLU9BMzlWfdBg\n8s9U8DCPp8eN8/9nDAy7wiVfi7MIjYEETAPmfJ51qBaBDiPAN0tSUuKmsK/mDUUjStqY7R7J\nfy1mlhjmY9FsLPWwz3xitR8X669ns7liLOqNVX8XE6g9LOzeDX083DQkvtgp49C2NhyXpOM7\nZf7Skb3RBtyg3hf64wcYHoKdR/ZWDuETtExRLQIdQMC/zplUkhIL7fT1/kmbF/rZGIvBcWbz\nMWI235+1Hcax7fLbZu3Qt94cFquen/p7jEAZCfgiMP2Ak6VlM8YtP0zAPG1bLThlPKmq7Ggc\nsqM50Loz0BNVJFYeLfMMQLXgaHqXqrKjsQ32TabyNPSECRNafSbCwqsWgSII+ETEZBQmJNOt\n5hp8u96azNfmCf3ZH/aFY8J+r5tvGJe697P+mM36VItAwwTK+OD/UcOra+FAHHXegWRX/fUh\n6EcgfHVtvCUI/bxVqFrgc1tVydb4LdwONleMP8raVuNol7c27WQ6fB6sJG8zqRaBTiMQJijT\nrfYJz9rss3bS9lp/LI4fY360Wdtq8/N6UjzvY23ztTihbnbVIpCbQBkJOGlRr0EHnxx1dEX4\n3TGPIG+FMPGxZoL0p2uhNlz6MLKagJFsp0Hn0flAwRExj9L9hVkPITHW3NuMJL0v/KrM1q1b\nt9Y/ThIxv4UYf7s9Iv6VHj787WefffYFixYtusdsONp9p7Ur9R2BLlUEOoVAUjIKk1eSX2w7\nOZb+FoM+vk2dJWbb3rPjr81rvqabRzhPkj1pvPmrFoGGCFSTSUOjGxs0CcP44I4TI8MPg43C\n5MjCJHwm5DdUmin4JaKrkTs/haTIK4+ZHI/HLUAX4wrphWgfiYuyLvDxcWR6JXR74w104b7d\n6+HL9Q2U8ePHcxuurajD/vKXv/zns571rEfhs0/FNgpjfoZ5ZsL2W8z1N6g/b/6soX/V62qL\nQIcR4HskTGxpm2BJz95bfqzZON78ssRK8klam83DOfz8jOP7kuLKLgItIVDGd8C2UL7QPwjh\n0W0s+Zqfr3mF8K8gX4A09bu5laNQJv5qQfI7B1dI34Xk+20Y97YOJN/HcVXzhaZnrStXPPOB\nJL4cgHm+D8PDmOdraO9qnZjnp/iZwitMVy0CHUSAicqSVbhsn9iSfMIx1MNk2IiPn5vj88yf\n5BvGjK1LNhHITaDMBMxbb74M2TnnKnnEeh6ER6RNlVWrVp2LpLesTpC1OFI9ZenSpTUXbdUZ\nU+3GBV/fxPh3YZ51VWO8wdPbPO2tIgKdTKBe0oz1exuTniU+1r7Pt42R+VL3bd9vdhvP2tr0\nC3Ub62vGsDjerrYItIxAWQmYR7KfaHLVb8H4pu4f7uvr24gEeTrinIME+XCwnn7YliN5Hrtg\nwYIbgr5cKr47vgpHui/GoF9Aqlc8Mwjm+D/IZ/BkrOOwlvtpUxGBDiSQNYklbRrHs4RxsiS9\ncMz2SMl/02JaomXt4/p2cmT1iEATBPgiK7rwe+b/hjw/YaI/wX4v5EHIXpDJkP0hsbU9CftR\nkDB5wpS/4Pd/92PChazBKedb/QVV+aPFR0ybNm0Mflf4r/D98z54XvTtSO6r4Zn2gRAP1JyV\nT8JaARkLqfmHoLmwGi0C1deyvV/ttU3dkloSJt/vx3l/b4/5m6/Nl6TTbms0n6y1X0PWMfIT\ngbYgMBOr4As4lBth41FirBwI4zLIFkg4Tt+ZAkrOwgRMjv4q75wh5N7jBOx9aBios4T27dYd\nf/P0J/nWs7OfxfxCfXvv4L/mP7hHFhHoEgKXYzvshW71wozbxlt2bIzVD2QcK7cdBJSAd7BQ\nK07A3l/x3nRr2thYn7dZ22qbyeu+bf1W+z5rs85SzD+Lr3xEoCMJ3IJV2wud9e0QngrNWvjs\nZD+e7WdnHSy/AQJKwHohGAF7L5ludZKd/exLKjbOfExP8qfdfGK1jbN4plsdjjE7a+vzNrVF\noG0JFH0RFr//5ROnfPkOlI3eUKcdu/qZF3WpiIAI5CfA70Fj34WaLSnxcSbfFyY7Gx+uyPx8\nHfpSNwnHU/fzxvrNlhbDfFSLQNsQKDoBT8WWht878og4T4n5H5kngHxFQARyE7CEyYFhwvQJ\nMUx6SXoYwy/I5grj0sfbqPv4aTHpGxabJ7RLF4EhIVD0k7Bi98LyCuc8Jeaf+WcC80wkXxHo\nYQJMTlkSmiVE+lpCi43z8fyYELHFMbv5ms469Altsfn9eLZjcUMf6SJQKoGij4D/iK3hlcy+\nvNwrGdoxf12IlQGcXEQgB4EwiVEPbRYuyW79luxYU2KxLIb52thY7X2SxtlcsfG02Tirk/xk\nF4GuIrAKW2NvDtZbIW/MuIUHwI/3CfvxbNOukp2ALsLKzkqegwnwPcdi78Ow7XXzDW02Nq2f\nY3yJjWG/2c3XdKvNbnWS3fpVi0DXEuAzle0NYDUf83gqJO2/USYNPq7Rxlh9E2wq+QgoAefj\nJe9aAnzvsdh7MKnt+wcGVP6Y3WrrM91qs4d11v56fmHcUG92fBhPugikEkhLgKkDc3QeCt/f\nJ/jzAqsfQO6F8ElYe0CmQPiAjjdBYmUGjItiHbIlEmACXgHRk7ASEakjAwEmKCv+s4N2r5uP\nr2NjzRYbG+szm8W1cWY33fpZp/V5P7VFoGsJXIEt4xuhWfkDYlR/zL5rabV+w3QE3HqmvRTR\n3rfhNof2UPf+1ufrsN/0NJ9YH22xEvON+ckmAl1NYBds3d0Q/4bI2+7H+L/uakrFbZwScHFs\nOz1yUvLy22XvVbPZmCS7+bE2H6tjfTGb+VvtfXw7rd/6rPbj1BaBniLAH1G4D2Jvhjz1Zoyb\nDVFpjIAScGPcunWUvfe4fWxnKX6M9zd71jh+rLUtBnVrh3U9X+uP1RYr1iebCPQMAf4Y/Tcg\n9obIUj8E/5f1DKFiNlQJuBiunRrV3nd51m9jrLaxXvdt669X25i02mLQhyX03W7VXxEQgUwE\neCp5KYQP1AjfTNR5q9L1kNMhPH2t0hwBJeDm+HXbaL7H0kpSP+1eGMPr1k6LHfZxTFgsjtXW\nb76h3fpVi0BHEYhdNVjmBozDZLzqeRJkImQNhA/v4BXRT0JUWkOACVhXQbeGZadGseQVe8+z\nz9tD3bbZ7D6Wtc3H4pjddPaH48MxpqsWAREQga4hoCPgrtmVDW8Ik58lxTCI2dN8bEzoY7rV\noR/tLL7f2ta33WP7X+vzNmun9ZmPahHoGAJFP4qSID4CeTXE/ydMu4oIiEB5BPj+S3oPJtmz\nrs5i+zi+zTjmwyRqbdotqbJmifVt76n9a+NqrdJEQARqCFwHjW8W3sP7D5C9ISrlEtARcLm8\n2322ViUvH8faVscY+D5r+zo2Jslm45L6ZRcBEQABS8D2htkE27cgJ0HC/5JhUimAgBJwAVA7\nOKS9F5M2oV4/x5lPrE6L6/v8WG/P27Y4ecfJXwS6nkCYgO3Nwno15J8h+3Y9haHdQCXgoeVf\n5ux8XzVS/Di2s+jmZ3WWeb2vtVk3UyxOMzE0VgS6kkBaArY3Dh+08V3I6yFlfC/dlaBTNkoJ\nOAVOl3XZeyrrZpk/aysxW6zPxph/6BPTzTes6WvxYuPMllZbzDQf9YlATxH4CbbW3hhZ6vvh\nfwHkgJ6iVOzGKgEXy7fdotv7rN666OdLbFzMZmOsr14c87NxrMMx3mZ9sXE+RtjO6x+Oly4C\nXUdgPLbo3ZAfQ7ZA7E1Sr+azn78P4a8ijYSoNE5ACbhxdp04Mum9FW4L/XyxcbRZO6n2Pj5G\n2LbxWe0+btLYMJZ0ERCBDAT2h8/HIHdA7M2VpebjKD8FOQiikp+AEnB+Zt0ywr+/krbJfNhv\nbV/bOG9jO6mYH/vDto0xeyxOzGbjVItA1xAYyquQXwSK74fw6Hj3jET5iMqfQPhdcVNl+vTp\nB40ePfrY4cOHH4NAj27duvXm1atX39rX17exqcApg2fNmnU+uvejC+Z9ZN68eZ9PcW9lFxOw\nnoTVSqLdE8snO/s8MBt1a/stDu02zvtYm+OtP2zTx/rMX7UI9AyBdnjx80fieZr5NAhvTRoN\nqVcaXjcS74QxY8YswQRM/GFZs2XLlmkLFiy4MexoVp8zZ857EOPrFmfbtm0rkYD5C1FlFCXg\nMih31hw+sdZ7P3lfv5Uc55Oq76vXtpj15q4XR/0i0LEE2uGKYx5x8r7gN0L2gZwB4Xe/tLe0\n4Ah0Hxz1/gpBY8mXc+09YsSI6+B3ZisnRvLdDwl3bitjKpYIZCDAJGeJLsk9lgDDMd6HbROL\n6f2T5gztYQyLZXXob/Yy6qGcu4zt0xxtQqAdErBHwaNh/kADL9xq+YVXOO17EeRoPyHaT3kd\n/aORhOfhSHmKtzfTRvJdirhZT7M3M5XGioAnkJbk2OdLvaQTxgr9qbOEcbdba//aWKtre7dr\n4XwxH9lEoKMJtEMCfhYI8qj3BsjDkPmQ10BGQVpWkFAPRxLkaeBqQWKcdfvtt0/csGHDZLT/\np9oxbNhYHCl/3OkNN3E0fTbmfV3DATRQBIojEEuWlkitttmTkqUlStZJPhYjax3OnXVcq/xs\nm1oVT3FEIEpgqBIwj25fC7kC8ihkKeQVkNgHAszVsh6ty6pajgYS6ju8OxLuCnwHu2D58uX9\nS5YsWY2+j/p+JM23Tps2bYy35W3PnDlzMuJ8Me84+YtAAwSaTX7he8/rPiFam/3exy85Zvf+\n1rbaj1VbBHqGQNkJmBcdfQHyAIT3BfO7WJ5urlfugMOHIbyN6TRI7oJEyAu8fGHyrxYk4xuh\n+NPRu02cOPG4qkP+xoiRI0cuw7w7V4byGdgqIlAUgTCZZUnIST6MxeL7zba9J/43XEPcq741\ny1z1o8hDBNqcQBkJeE8wYPK8GXIb5DzIwK04qNMKL8LiVcM8Mj4S8mXInyGNFsbw5W6voM0P\nm3u8bdSoUeEY353axqnnc+HwcnPCEfe/Wlu1CJREICmR+cSaZylJCbbReHnmDn2HYs5wDdJF\noCkCZSTgb2CF/wF5QcaVMjHyN4R5tPteCI9MmyozZszYCwF280GQEPlwj5oS2qAfVuOQUTn7\n7LOfiwu5Puvcr8R9xt9xupoi0CyBZhKQJVKrk9YSS+A2L2sr9eKYX1hbrNAuXQR6gkBLL3Rq\nghhPz34XsghyfRNxokNxGnjQFcibNm3aEDrDL7TVJO3QP6afcMIJo/B981fRxyu6hyGJP4qL\nvObg3mOeCVARgVYRiCXHtNiWMG1cqHMs+ywpmp/VabGtz2JSrzfOfOv5WeywbnRcGEe6CAwZ\ngaFOwPdiyxdDLoWsLZCCfQ9bnQIJkRd01RQkyw1Iwt42aJzvjLWPOuooXj19rPUh3tlLly59\nAhdktTIBPxvxL4RkvUislXPbpqluPwI+qdW8kFOW6sfQrd4438+xXrdpYjb2hXOZv2oR6EkC\nQ5GAN4P0f0EWQq6D2JsSzWIKkuDA0aiPDhvXUVNg668xbL8nOTAlq0iyTLz/7Dy+OnfuXG6r\niggMNYGkpGjr8u/DmK/v55iYT8xm8cM6j284VroIdAWBMhPwfSD2FcglEN56VFrB96/r8Z1s\nzXybN2+OHT2GiXpdzaAU5bTTThuHq5556nmAKY6mH37mmWd48VkR5QkEnZ4j8Evg2/Tzs3PM\nJ9ehIZAnqXlfS67e5rfA+r3N2uwLx3l/9pke+lkM1SLQkwTKSMA/Bdl/h1wD4Y8pFFFeh6BH\nVAI/g5qntasFiZG2moKEHCZbfl87DkfB3u9pr6S1d9ppp8+g/7nmg6T/gWXLlj1pumoRKIBA\nLPn5aZL6w4RoL/rQ7mP5tvl7m7WT5rR+1SIgAhUCZSRgfldZdOHV0n9XmWQN6poEDP3xSl+1\nQrKdUFV2NEJbnu+lz7IwiL0NCf7K2bNnm4l1zSE4Ev0R6LcE/W+4D/nz3lltEchAwBJhUuJk\nf1JfhvBVFz+Ptdnp2+ac1Wb+qkWgZwnUJIVupYDkxgTM07bVgnt8J1WVHY1DdjQHWncGeqKK\nhFr94GEbZddAdgkGj7B+2AcdjQe+UruXABOkJclmt9LiWM14fF1WX5uVCcxmc1td6U6tYr5+\nvtTBkc5YvIibTCLQfQR6IgFzt+GglE/Tqhbodsp6wIZnRY9GQpxSdUADPnxwiIoItDsBS6iW\naK22dfsk59vWn7UO57Fx4XxmVy0CIpBCoIxT0CnTl9rVh9leZjMi2U5D+yLTcUT8AbT9hVkP\nzZ8//3fWzxpJel/4VZmtW7duLb7ntXuHebVzeAq7OhzJfBfMeWLVsP2xl9dW9LucXc3eItDK\n5BUeiTJ2aIvRrbeGMEY9/9gcSbZWxkqaQ3YRaEsC1WTSlqtr4aL6+/uvRu78FJLgSIZFfTy+\ng70YF0stRPtIfGd7gZ8OCfNK6DUfPHjAxvXwPcz8xo8fz4Q6kERxu9GpZo/VuEXpMFwMVk20\niP8gTo2fEvOVrWsI8PVjCcZeS6a3YiMtvo9tbcb3bZvXaj+/xfG2WDs21vvZfPX8/Bi1RaBn\nCfTMKehFixbdg71cc3EWkuk5TIpIvt9G3972KkByfBxPryrj4jGbUrUI5CVgyc6PM5tPgGx7\n3ftb2/dbDOsz3fuwz+zmp1oERCAngZ5JwOSyatWqc5Fcl9VhtBZHxafw6VV1/NQtAvUI+KTF\nttfrjc3T7+OybcnR2xmPdhPq7A99Qp1+WUssXtax8hOBniPQUwm4r69vI077no69fA4S8cPB\n3u6HbTmS77ELFiy4IeiTKgLtRiBMdl73bVu3JWXTs9YWyydujqVdRQREoAkC3fIm+hoY+PuA\n98nCZM6cOfsx4ULW4JTzre6CqizDO8mHT8JaAeHtTps6aeFaayIBS6hZ38OhP3WODe1JE5qf\n9cfmNZ9Yn41TLQIi0GUEmID55qeU+pjLDuHIBEw2/irvDlm6lhkQsNe51UE37tL/AAA1Y0lE\nQVR3ZtXGh3VaAPNN8qnXnzROdhHoSQI9cxV0T+5dbXS7EGBiavaokDF8sXhJdu8ba9t46wvj\nmO79fNvG+bpev/dVWwR6nkBPfQfc83tbAIaKQDskJkuoVocsuMZ2WGe4Luki0LUEdATctbtW\nG9ZBBCwp1kuA7Ddf27x6Y8wvb50U1+YP+5PseeeVvwj0DAEdAffMrtaGdgABJjFLZFyuta2m\nLUx8tFnxfmaz2sZZbXZfc3xaDO8bthk3LXboL10Eep6AjoB7/iUgAG1AwBJXLPnFbOZvSzef\n0G79Wess47P4ZJ1PfiLQ0wSUgHt692vjW0CAya9VSSkpTpLd5vb9eZNxXv8WIFMIERABElAC\n1utABNqDQFoitESbZaU+GRfhnyWmfERABDIQUALOAEkuIpBCIG/Ci4Wy5Bvroy1pjiS7j2Ox\n6Wtt688y3nxVi4AItJiAEnCLgSqcCKQQCI9kw4RoQ5Ps1p+nbibJhuvNM698RUAE6hDQVdB1\nAKlbBFpMIJZck5Ik7Ul9XBZjxeKxL63Ui5s2Vn0iIAItIqAj4BaBVBgRAIF6R4w+mVripM3a\nhFgvBn0aLX7+LDHy+meJKR8REIEKASVgvRREoHUE6iUsn2i9r7V9f71VNZKoLb7NV28O9YuA\nCBRIoFsS8CIwuqbCaUOBvBRaBJohwMRnSbCRBNrM3BwbS7y2nlhfs/NpvAiIgAj0PAH9GtLQ\nvQSY4CzJ2Sq8LaltvvVqP76eb6y/2fGxmLKJgAhkIDBUR8B7YW1vgkyFTKnUB6D+I+ReyB8q\n9U9Rr4aoiEA3EUg62kyyh9vOpMlC/3pjvO/AoOBPvfGBu1QREIFOJTAeC/8nyFMQ+887rd4I\nv4shz4aoNE5AR8CNs8sy0pJcFt/Qx17/oT1NzzNfI/HT5lafCIhABxI4EWt+AGIfCHnqP2Pc\n9A7c5nZZshJwsXsilhD96zs2u++PjecY84mN97asfn6M2iIgAj1CgKea/wKxD4pG6q0Y/4Ye\n4dXqzVQCbjXRwfHsNW09prP2JdR9X9i2GLT7dpJfLHbauDCOdBEQgS4jMA7bcwvEPgiaqZ9A\nnEkQlXwElIDz8WqFt73Ow1i0t7qkxUxaR6vXoHgiIAI5CZRxEdb5WNMxCevqh/0hyIOQRyH8\nrnd/yIGQnSBh2R2GxZCTwg7pIjCEBCwB+guafNsvLcnuffK202Km9eWdR/4iIAIdRuBnWK/9\nF271Oti+BNk3YVvGwv4BCK+GtjFWr4dtNEQlOwEdAWdn1YynvUbzxsgyjj4qIiACIpCZAJ81\nHV7xzKNeXpCVpfCIeBXEPqCsPjbLYPlUCSgBV1EU2rDXZ95JsozL4pN3XvmLgAh0MYHDsG32\nwWH1F3Nu72siMXRFdD6ISsD5eJm3vWZND+t6/aF/PT0tXlpfvbjqFwERaEMCRf8a0jORbebD\nNfKUX8B5czDgkUCXKgJFELDvT5n8iiwWn/PZnJzPJ92wr8j1KLYIiEAJBIq+CItPtuJ3tnwA\nh5W11shYM/nytLX/3vdXGccmuk2fPv2g0aNHHzt8+HBeIPbo1q1bb169evWtfX19GxMH5esY\nPnv27BdjyKGYY8q2bduewRy/h6xctGjRPflCybtNCPhEySWFyTK01Vs24/kY9fzVLwIi0EUE\nynjzXwZe73PMzkabVzJnLcfB8b+dMy/Meo7TczWReCeMGTNmCQa9OzJwzZYtW6YtWLDgxkhf\nZtOsWbPeNmLEiE9iwFEJg67asGHDR5csWcIrwMsoPAW9AsKL2zaVMWGXzhEm4FZvZtHxW71e\nxRMBEWiCQNGnoLk0Xu3Mo2ArH0YjdouR9fuaR+hf8Aa0Px3omVUkxn1w1Muj51jyZZy9kTiv\ng9+ZmYMGjnPmzPkgYnwb5qTkyxHvHDt27B04Qj40GC61vQnwH9bwn1YmTUuc7b16rU4ERKCt\nCJSRgH+HLT4FYt/jPhftKyB7Q9LKvui8HHKCc+K4rzo9VxOngi+CHB0M4lXa1YL+0Uig83Ck\nPKVqzNiYMWPGS3Gq+aKI+1Ow13xIY55d4Pd1zONPrUeGytRFBGpeA5HtChN8Pf9ICJlEQAQ6\nhUAZCZgs+iDvhfBxkixvhvBXjy6GnAp5OYSJ8XWQD0AWQVZD3gWxcgcacyEvrCPPswG+RqI7\nHEnvPd6GnDjr9ttvn4jTwZPR/h/XNxZHyh93eqbmyJEjz2UCd8534DvfaWvWrNkTtkMwx+dc\n3zD4vnDUqFHcZpXOIBBLiGHSbOWWhEfbrYytWCIgAkNMoOiLsLh5l0COr2znRtR2QRZPQ59T\nsWepeOS8IoPjw/A5IPRDQn2HtyEZrpg3b94C2pYvX74ap4M/iuYN5oPk+NZp06adffXVV2f9\nzhRDhv+NjWeNOc6aP3/+Lyu2+1H/M+Z5D/wmVWysXgD5vtPV7HwCTNTDIZaw2Waxers2+G/o\nP9hDFhEQga4hUMYR8MGgxeRJseRbJED7EKuZA0nvpBrD9tPgVROS8Y1Q/Ono3SZOnMgLwDKV\nmTNn8giX4/lPBpPv04gZu1r72z4g1nWI19VuawL1Emi4ePrnHRPGkC4CItClBMo4Am4XdEcG\nC7k70Jm4eXtQ9SlbOD3MMb8I/KIqrpzmE7sGkumZZ575bJyO5nOrY/8MTPABkKjv8rraDREg\n56ITne3LLPNk8YltaKPjYrFkEwERaHMCPZGAcXHUXtgPu/l9gcT3kNfZpg1HpNUEDP2w0CeL\nvnTp0ifgR6kpOKXNMwB/W2Pc/ktRgUlqAQTyJNCk6Zkgy0j2SfPLLgIi0EUEyjgFPeS4kFR5\nNFpTNm3atKHGAAV+oa0maYf+efW99trrM5hjHxuHBP8o2jeZrrphAkUcOTLRWtLmwoqYo+EN\n1kAREIHOJ1DGEfA/AtOzS0QVJlFOvXM4Px7GsT60ISFuQIL05kHjfGeeNi6++hBi/70fgyuk\nz8Op66e9LWObPC+EjMnoz6uwe73U7NgmYOSNoyPmJmBrqAh0M4EyEvB/DzVAJD4+AaqmwGb3\nJVftsPVXle2NcYHekIrkew5i85arakGy/zmS79eqBjXajUDeRNvI+u0Iu4y5GlmfxoiACBRI\noIwEXODys4XGkeZ6PFyjxnnz5s2xo8cwUfN3i5sqSL4fQ/Ll0aovD+CRlzX3JPvODG1+vzw9\ng5+5vASN15vShfVQJrJ6R7hKrl34gtMmiUArCNRmpVZEbMMYuCL5mXBZSMhhsuVFWOERbyOn\nh6tTIfn+Yyz54h+CVy9cuJD3K6u0hgCT3FAkOkv83Aq2vU5bvWJrzjuuXlz1i4AIdACBnjgC\nxn54PNwXSLY1twNV+kPb2nBcVr2SfD/n/THnH6C/Bg/nuN/b1R5SApb8LBnmXYyNszrv+DT/\nZteWFlt9IiACQ0ygzCPgkdjW0yFXQe6E/BHCI0xeNJVXMCR7wQMxmIBrbgvCPb6TIhEOCWxc\nZ+6CH2T4KI58w+R7KxLwy7GW+3MH1IB2JdCKpMsYaXHS+tqVi9YlAiKQgUBZR8D8YYNvQl6Q\nYU2FuCD53YGk+DILDv0ItH9kOn8UAf01P8AAn9usP2uN5Pta+H7e+yPOb3Ha+TW46OrP3q52\nWxBQgmuL3aBFiEDvESgjAe8KrD+AHD7EePswfzUBI9lOg36RrQlHxB9Ae4zpqB/CqWL+klO1\nIEnvC78qs3Xr1q1dtmxZ9bYn9B+EZPt1xPZnFtZs3Ljx9fjtXyXfKsnSGq0+hdvqePVA6J+D\neoTULwIdTKCaTArcBv7gwlAn32H9/f1XI3d+CsmRp8L50I3j8T3txTgyXYj2kbgo6wLPAIn0\nSuj2gTvQhR90uB6+h5nf+PHjT0T7WtNxb/Gn0Z5oOmvEGYXf/v0p5vLmahvxrp07d27N/cHV\nTjWaJTCUCYyvnaGcv1l2Gi8CIlAwAX+kVtRUZxQVOE/cRYsW8TnPi/0YJL9zcIX0XUi+/IGE\nva0PSfNx/ERheOuQdUdrHv2iY9CtRZhjIuToJMGYg6MBZcxCgEmu5p+khEFZ/RKGV81MqFmT\nala/anA1REAEeotA0UfAfJQjE1NYeFvQDZBHIE9B1kMKL6tWrTp38uTJ45EMT0uZbC2OiqdV\nnuec4lbbhaPjN8FSNM/aSaVlJdDqZGhJv9Vxs26P/ERABLqAQNEJY/8IIyZePhSi6YdcRGKn\nmvr6+vhTgafjQqlbcJR7HhKxX18/bDdB3oeLpR5MDRTv5MMuVMol0MoEmCeptnLecolpNhEQ\ngbYhUPQHySHYUv5Mny9/BaXm4ibfWWYbiXg/HO0eC1mDU863+guqylxHCXPxn4MVkLGQTSXM\n14lT5EnAnbh9WrMIiECPERiN7eXpZfsOjjWvilYplwATMNn7q7zLXYFmEwEREAERqCFQ9EVY\n/MGD/6yZcdiwkwJdqgjkIWD/zOUZU7RvO66p6G1WfBEQgQ4gwB+474fYh9RP0NaRWLk7rpuO\ngO11VC7B9NnacU3pK1avCIhAzxD4JLbUPqRY3wwZ8nuDe4b+sGHdlIB7aLdpU0VABLqZQNFX\nQRu7f0WDPyI/p2LgIyl5IdatED7u8UkIb03KeoEQH3ih0lsE+I8bS9EXDm6fRX9FQAREoIsI\nzMS2bIX4I+FG212EpZRN6YYjYHutlAJMk4iACIhAtxD4EjbEPkBbUXcLl7K2oxsScFmsNI8I\niIAIdA2BN2NLWpF0fYyugVPShigBlwRa04iACIhAVgJF34bEdfx91sXITwREQAREQAR6hUDR\nCXgngHxlr8DUdraUAM94qIiACIhA1xIoOgEfECG3EbZ/hBwN2R0yHsJ18OrWrAJXFREQAREQ\nARHoXAJl3YbkCfGH7y/3BrVFIEKg2duN/BF0s7Eiy5NJBERABJojUPQRMH9VyH8QcrU/aW7J\nGi0CuQgo+ebCJWcREIGyCBSdgPmTgzcGG1P94fvALlUEWknAvs5oZUzFEgEREIGWESg6AXOh\ni4LVvjHQpYqACIiACIiACBREgKedeSqa8gjkFRCV8gi8BFORvX4EozzmmkkEREAE2oLAnlgF\nn/tsSZi/jvRvECaG3SAjISrFEVACLo6tIouACIhAQwTKuEDlDKxsCmQi5OyUVW5AHxNzlrJL\nFif5VAkwAa+AjIVk/cGL6mA1REAEREAEOpPAdVi2Hfm2qu5MEkO3ah0BDx17zSwCIiACUQJl\nXIQVnVhGERABERABEehlAkrAvbz3te0iIAIiIAJDRkAJeMjQa2IREAEREIFeJlDGoyivAuDf\n9DJkbbsIiIAIiIAIiEBvEtBFWL2537XVIiACbUxAp6DbeOdoaSIgAiIgAt1LoIxT0G1Jb/r0\n6QeNHj362OHDhx+DBT66devWm1evXn1rX1/fxlYtuIw5WrVWxREBERABESiXQBkP4ih3i+rM\nhqQ4YcyYMUvg9u6I65otW7ZMW7BgQfgDEhHXZFMZcyTPHu3hKWg9iCOKRkYREAERGBoCPXUK\netasWfvgqPdXQB1LvtwDe48YMeI6+J3Z6O4oY45G16ZxIiACIiAC7UOgjCPg87C5z23xJp/e\nSLzZs2d/Haec3xOMfQr6swLbxk2bNh25ePHiewN7XbWMOeouYrCDjoAHM5FFBERABIaUQBnf\nAb8OW/iqFm9l7gSM08KHh8l327Zts1auXPmVqVOnHjh27Nir0H9cZZ1jcaT8cbRzzVPGHC3m\nqHAiIAIiIAJDRKBnTkEjob7DM0byXTFv3rwFy5cv71+yZMlq9H3U9yMZv3XatGm5fr6vjDn8\nGtUWAREQARHoXAI9k4CRUE8KdtMVXkcy5oVXPB1tZbeJEyfaEbHZUusy5khdgDpFQAREQAQ6\nhkDPJGDskSODvXJ3oPOXmu7xtlGjRoVjfHesHfoXMUdsXtlEQAREQAQ6jEAZ3wE/Aiar6nDh\nPwIjIeMg/K1f1r4wOX4J8rg3Zm3PmDFjL/ju5v1xCvohr7NNG45ijzU79MOsXa8uY456a1C/\nCIiACIhA5xAoIwG/twEcTJYvgyyC7Afh1drHQ14D2QjJVZBUdw8H4CrnDaENfqGtJmmH/l4v\nYw4/n9oiIAIiIAKdTaCMBNwIoScx6AcQHo1+D8Lky4R8OeSdEB4R5yk7h854GMf60IYj3g1I\npN48aJzvDNqDfAuYw6Z8NhoXQrJeJLanDVQtAiIgAiLQHgTa/TvgR4GJtwNZmYbGP5qStUZS\nHRv6wrY5YusPbOGp8KB7h1rGHDtmy936C0b8GrIp90gNEAEREAERKIRAux4B+429AQoTyK4V\n42zUn4dsqeh1KzzneT2ecFXjt3nz5tjRY5io19UMSlHKmMNN/wTa052upgiIgAiIQIcRqM1K\n7bl4Hqn6q4n5nfCL8yx15MiRz4T+SMhhsuVFWOER79PhuCS9jDmS5pZdBERABESg8wh0QgLm\nke/zA7STA72eOujqaSTbCZFBoW1txCfJVMYcSXPLLgIiIAIi0GEE2j0B89akCyDhqXJehJS5\n4CEbTI48bVstuMd3UlXZ0ThkR3OgdWegJ6plzJE4uTpEQAREQAQ6jkCY2IrYgOchaJbbeXj5\nMRMuTw3ziuJ9Ie+B8ArosPwxNNTTccR7By6U4pXUAwX6EWj8qKIOw3OcR6N/iums4XOb1+u1\ny5ij3hrULwIiIAIi0BkEykjAFwPFq1qMI1dirMzdh7qagJFseUX1RbYuHBF/AG1/YdZD8+fP\n/531s0aS3hd+VWbr1q1bu2zZMn/vcNNz+PnUFgEREAER6F4C1WTSQZvI22nuyrve/v7+q5E7\nP4XEy6PsYaiPx08HXoyrlxeifSQuyuKp7mrB0eyVUGruN8aPLVwP38PMafz48Seifa3prZjD\nYqkWAREQARHobgKdloB561Hu+4C5CxctWnQPEu5iNGdSZ0EyPQdXL5+zXdvxF8n38Q0bNvBB\nF7lKGXPkWlDzzqMRoubJJM2HVAQREAER6AgCvAOn5iCs1avupARMGB+GXN8ohFWrVp07efLk\n8Ui8p6XEWIuj4mlLly6tuWgrxb+mq4w5aiYsTuFXB4P+OSluOkUWAREQgbYiwBwwscgVdUIC\nfgwAfgL5d0gj3/1W+fX19W2EcvqcOXNuwVHueUjE+1c7hw3rh+0myPsWLFjwoLPnapYxR64F\nNe58P4beCzm18RAdPfIUrP6t2n7t/45+FTe+eL3+hw17Y+P4so0s4/Ri1qug/Yp5qvnPEP4H\nsgZSyGkAJOL9cLR7LGQNTjnfGlxQhWmbL2XM0fwqoxHmwHo25Ohob/cbtf3a/3r96/1f6Odf\nGUfAt+Ozems7fl7PnTuXtzPlvqUpz7aUMUee9chXBERABESgPQiU8SAOPrt5JeSfIJMgKiIg\nAiIgAiLQ8wTKSMAfBOXnQj4LWQW5CdLq+4IRUkUEREAEREAEOodA0QmYT8B6jsPB75z/GpLn\nGctuuJoiIAIiIAIi0B0Eik7AfJxkWPgQjaauZg4DShcBERABERCBTiNQdAK+G0B4NbMvvJ9X\nRQREQAREQAR6mkDRCZi3E10TEOZtSZMDm1QREAEREAER6CkCRSdgwjwf8lBAdT70iYFNqgiI\ngAiIgAj0DIEy7gN+ADRfArkccgKE5bUQPm3qe5AVEPr8CfI0hL8uxCNnSlK5L6lDdhEQAREQ\nARHoBAJlJGA+QvL5ED7Nikl14NeIUI+HvLsiqHKVMp7glWtBXejM7+o3deF2Zd0kbb/2v17/\nWd8t3efXNe//67BvmHxbKd23u9tvi8ZiSQe037JKW5G2X/tfr//S3m5tN1HXvP+VgNvutaUF\niYAIiIAIDDWBMi7CGupt1PwiIAIiIAIi0HYElIDbbpdoQSIgAiIgAr1AoIyLmV4DkPu0GObX\nWhxP4URABERABERABERABERABERABERABERABERABERABERABERABERABESgGwiU8R3w6wBq\nPwfrD2j/3On1mnyK1qshvCeP8hjkDIiKCIiACIiACHQsgTISMO8DfpUjxEdSvs/p9Zpnw2Gh\nc+pHey9I+CtLzkVNEpg2bdr4PfbY45jhw4e/AOqeqG/ZsmXLbxYuXPhwqwiVMUer1tprcYZi\n38yZM+forVu3nmms8Xq7dNGiRb8zXXV5BMrY/zNnzpw8YsSIY7Zt2/YcfL7sgvpebOHda9eu\n/fXVV1+9pbyt7cyZOiEBHwm0twd4T4b+48Am1RGYPXv2iVCvxJti0I9e4E1y6apVq2b29fVt\ndENyN8uYI/eiNGCAwFDsm5NPPnnslClTfoMFHGW7Acl42vz5879luupyCBS9/2fMmLHXqFGj\nPoetOR0y6HZWfMbchn++zsE/+z8rZ4s7c5ZB4NpwMyZH1rRvxCZThQCOQj6IxPvjWPKlC+yn\nT548+Ua8SQcl56wQy5gj61rkV0tgqPYNku+nsJJq8q1dlbSyCBS9/88444xdRo4c+UtsD890\nRHMIPmOORoK+Dp8xbylruztxnlb9GMOXsPF7JAB4bmB/KfSvBrZQ5ZE517YzhN//huVZoUH6\ndgJnn302eXN/VN8Y+G90G94Q/wdblRv042C+ELYPQHKVMubItSA5VwkM1b6ZNWvWX2MRH60u\nRI0hIVDG/p8wYcLF2LjJwQY+hc+T+5l4A/tSHC3/EkfCvHZHJSBQ/ZAO7HnVuzHg1AQJH8LB\nHZfka/b3wuddkDdC+KtJYfl9aJC+ncDo0aMvQKv6jxXeFLds3rx50qZNm/gPEr9P9+VMvDkm\neUOWdhlzZFmHfAYTGIp9c+qpp+6ED97LsJpWfZ4M3jBZMhEoYf+PwGfKO/1ioP/TmjVr9pk3\nb97zYKc8YP14XUzE0fJrTVddS6BVb5jFCFvmhRa31G6GNBKYPn36BFT8p6Va8Ob42OLFix+A\nbJ47d+5i6NdbJ94cw/HmmGZ6lrqMObKsQz6DCQzVvtl1112/gJfS1MErkqVMAmXsf5xSnop9\nzTOTAwWfJ3967LHHLsIFV+tpwGfMbaguHeis/IE/f45WJUKgVQl4K2J/OBK/CNM1CPpoEYE7\nPSaSKU8D7mTbwTcHLoAhL19+4BW8OU7yer12GXPUW4P64wSGYt9ULvaZ5Va0ybXVLJFAGfsf\nZ9P4VZYvdyH5hvt8pXfA59CTXld7B4FWJWBGvAHyzR2hC2k9gqh5bmEqZBHtGhRvQF4x7svd\nXqm0Q1s4JjJkh6mMOXbMplYeAmXvm9NOO203/AN3ia0RH7S/hXzfdNXlEihj/+NMGj+Dq2c7\nsf//+qyzzuLzGaoFr4GaU9S4Ev6H1U41agi0MgEz8HmQP0Dud7IBbV+egeL7Y+374HMv5C7I\nbyDfhvwdhKe51kBUIgTwwj8sMD8U6MPgE9r25VWNoV+SXsYcSXPLnk6g7H2z0047fRkrsg/f\nTbjt5P3QN6evUr1FEShr/2Oeq9w2jBg7dux3cSbkDFx9fQLqr6Dvza7/rgULFvzW6Wo6Aq1O\nwA8g9nMgk5ysQNuX70CZVEcOQT+T7RGQ4yBvh1wBWQdRSSawu+/Cf6fhPz/D+vv7B9nGjRu3\nmx9Xp13GHHWWoO4EAqXtG3zQ/i1eX7xocqDgQ/mTuNL1dtNVDwmBUvY/Lrb6PPb3EttCvA5e\nCFkK/XrUZ0FGsw8+161fv553vWyjrjKYQKsT8OAZZCmTQPXiCE6KN8DAhRF+AXhqzaAEjP6a\ncd4/0q7xLWiOyLQyZSBQyr7hQxjwIbvI1oPXwP/gQpzPm656yAiUsv+xdVuRhM/Gfl+WtKXo\n+zP+2f/Q0qVLn0jykd3drlIgjIsQ+0oX/x7XVrO1BMb6cPiQHHQ6EG+Kftwg792G4TuacTWG\ndKWMOdJXoN4kAqXsG7x+mHz35CLwQbsB8n49djBpl5RqL2X/n3nmmc/GWTM+6IdnJ6MFfbvj\ndXILzpScj2T9H1EnGUu5b+9H4MzTFSY3iHthBGqOePHBOCacCUfANW9S9uPNkufUfhlzhMuW\nno1A4fsGD9zgRZBvdcu5AFfa3+l0NYeOQOH7H5s2fPz48d/xyRefM7/EP/GnQF6K9ifgs5EI\n4DMacjGS8LuoqwwmUHsoNLhflg4igBc7L3DzZVCyxY36g4528aZ52g9Ka5cxR9r86ksmUPS+\n4dWumIMXXg0UfvDi6IZnuFTagEDR+5+biH/A+Iz5V7rNXYPnyv+Ne678Cvjcg3/0v2E+WNdn\n8MMQOEmiH2cwJlYrARuJ7qgf95uBFz4fzFFT8KE5AXZv2/r444/n+Z6mjDn8+tTOTqDQfYOr\nXd+O186ubjnPx9FN+Nqpec3B/zL4LEG9DQ9pqLlIyMVRszUECt3/XCISKy+IrRYc9X7XJd8B\nO86IXIV9/iXs8/0qjlMmTpzIC2p1kV6V3PaGEnAApMPVmnt8kWwnhduDNwWvMK8W+NyH/0xr\nTl1VO+ONMuaIzyxrPQKF7hu8VvDy2fHPG9o8mzLojIpfJHwGEjLG6kpYD6aYdqH7v7Lkms8P\n2K5L2JT7YN/P+vA6OBhtJWADUql1FXQApJNVfMbdEaz/UJz6GRnYDg90PjoucyljjsyLkWMN\nAe2bGhw9p5Sx/zHHnwKw+wT6gIqEu6+3Y9wfva72dgI6Au6iVwK+i7kJPzP4NF78A7cjoN51\nzz335Hc2P+Zm8vdaUZ3Ftis/cO2BJp4puy+uYKy+NtatW7d22bJlA7cvtWqOcE7pzRNo1b5J\n2f9/wCp5H39aeRE6DzAHfPD+Gq/DhyFmUl0QgRL2P696/w32ZfWiKrTfi82ZB6me4cB3wPyl\ntUm2z9HesmHDBl2oF9nv1Q/ZSJ9MHUaA38Xgu5fvYtnVByTgTTAPT6g5F9/V8MKIj6DvQLdZ\nPPVM/5qCC7V4Q/1hZsRVj0zi11Jv1RwWW3XrCLRq3yTtf3y3x8dMpj5qEq+/K/HaqX5A48P3\ni7hQ61ut20pFSiJQ9P7nvNifv/DzY18fj31+Kez/gNfHY7hH/CX4nLkMPv4/rhvtH3g/Vm39\nfFg3vgZ4G0D1O128QSZD/0+8KXh6mj+gXS1401yID8eaCzeqnemNMuZIX4F6kwho3ySR6Q17\nofsfj5X8FT43au7rxWfM+/H58gj+0V+PE2c3AfMUQw3fDfjnf4bpqmsJ6DvgWh4dryGh3o9n\n8r4WL/zH6mzM13DKqqGnF5UxR521qzuBgPZNApgeMZex/1euXHkePl9+GkFa89wBJl8k51lI\n2r+P+MoEAkrAXfgywAv+RvxsGL+HuQ6bN3BTvNvMtWh/CLeEnMpTVs6eq1nGHLkWJOcqAe2b\nKoqebBS9/5cvX96PRP86/KN/Kj5jVoWQYeP3wdfioXvPw+fMpWG/9B0E/Hn6HVa1uoYAroIe\ngwuxjsEG7QO5Bd/TPNjqjStjjlavuVfiad/0yp6Ob2cJ+384HtCyPx5NORV5dy+cbr4XiflO\n/GxhnqfrxRcvqwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiI\ngAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiI\ngAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiI\ngAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAj0DIHhPbOl\n2lAREAEREIGsBPaB4xcizvNg+3XELpMIiIAIiIAIlEpgJGabAflSqbMWP9lzMMW2iLyj+Kl7\nZ4YRvbOp2lIREAERaCmBVyLazZAFkD1aGlnBeoKAEnBP7GZtpAiIQAsJHIRYV0GWQ46BqIhA\nQwRGNTRKg0RABESgdwlch02f2rubry1vFQEdAbeKpOKIgAj0CoHYgQu/L1URgVwElIBz4ZKz\nCIiACIiACLSGgBJwazh2chRexXkEZLecG8GjAH4XxrGjc47N6z4BAw6ryE4ZBnM9h0COhOTd\nrgzhh+0Op6MgrFtV+F7klae6mKdVRONxynzdxlcw2FrEvuctpvtC+B31AZAi36Oci6/dXSCt\nKrsiED9b+D7bH8LPgFaWRj73ithPrdwmxWozAidiPT8L5N8ra2Ty/AbkCYjdbnA32h+EJP1j\ndij6LoSsgvRDbNxGtG+BXA6ZCYmdooN5UJkOS7i+M5zXe9C+AbIVYnOxXg55CyQsx8LAbVoP\n8f73Q/8XCN/USeVgdIRr+Z5z3hPtz0Dug/jYZPEtyKsgeQv/qfh/kJsgT0Ms7kNo/xDyWch+\nkKTyIXSEa6Z+dNIAZ+e+jI19p/MpovkPkXn5nSr51ivj4fATSLjuj9QZ2IrX7evdvOHri/vt\nEdfP9b0Oklaa3fex2C+Bka/ZeyF8T9rrifVWyJ8gfZC3QeolZCZUP97adhvSOPRzXzLenyu+\nW1DfAbkM8mJInsJ/Zrkfb4M8A7H5fP0w7EshJ0LSCvvD10ijn3tF7Ke0tauviwi8F9viX8Bs\n/xTC/4gfjPSxfyUkLPzP9gpIGCtJ/y18nxcGieifj8RkkuOHMT9ok+Kb/UvwYeF/4J+E+H8K\nzMfXd8JnCiRWnguj92X78Yrjm1CvifR7f374cO2jIFnKWXBK+qDxcfmh+eaEgCfA7n2tzX+S\n6pVPwMH8fc0P3iILE5Ofz9ofzjDpuxPGvjphbCtft9xfttYs9ekJa6K5Ffveh58EhYkwy7rM\nh+9/Hl0mFb4OzNfXTMAHQn6T0G++/AdgFiRLeT+csrwXLDbrqyBJZ8Na9bnX6v2UhYV8uohA\n7IXI/wyvh/gXs2//U7D9fJPyyNj7ZGnzDfhxSNLRNLqGxRLwQthvhWSZgz4zITyKzOp/M3xH\nQsKSlIBPgWO9xO7n5jallfHo/BbEj8nSnosxY4LA/MdjVSTWvYFfTP1dZNyKmGOLbXw9rI7M\nzf1Sr8T+KbsHg8ghLK1+3bYiAbdy39v2MibPPmV5DYU+azEu6R/lpAQ8D2MeyzHfefBNK9PQ\nmef95bfh6oTAzX7uFbGfEpYqczcTiL0Q017sPE01yQHhKdv7IP5Fn7f9QRcvbH4+EptryDNH\nXn/Gjh2dxBLwZvjyH4k863kK/mmnui/IGc/P/TmMDUtSvBeGjk4/FG0f19oznU+RTf6TZ3P6\nmkkzqfB7QJ5l8P5sfywyoIjXbSsScNK+Crcppsf2PTd9MSTm/yTs/CfrFxD/9Ubo+2v0x0pS\nAg7H19N5an5MbALY9oSsh9SLkdZ/eCR2s597ReynyDJl6nYCsRdi+GLmm5Tf22yF3BQA+Qb0\n0J86fb8JYXLl90mfgtwOifn+H+yTILESS8AWg2/cf4QcDdkLcipkA8T6YzWPkPgfNT+Aj4Es\nh8T8roI9LLEE7MdehwHTIftBmCj4JiUH72Pt2bDHyoEwPgMxP6v/BNv7IAdA+F3YGyC/h1i/\n1fznidvly8FQYuv4gncK2rEESLbPDvyKUvdB4E0Q2y6r09bMRGt+VjPG3pCwFPG6fSUmuawi\nsYR2r+un38shvhSx74djAiZa42H1h2Eb7SYfhfaLIDdAzMfXr3S+1qyXgL8Px7+DkD+T4Icg\nScn03eiLlU/A6NfBNl/LF0H4tcIECF+Tfw3pg4S+1M+BhKWZz70i9lO4Puk9QiDthfhjMJjs\nODCpvMrpfNHHXvBPwP5G52fNsWgshMTGXGNOQZ2UgJloXhH4Uv0HSCw+bTdDwv+0+QaOfVje\nCXtYngtDUuxr0cdTp2H5HAyxMQtCx4oeSwx/Qd/BEf+dYLsbEsZfFPHl+kK/1RE/M/024v9t\n6yyp5unDcM0Pwxb7eoBL4j+Jof+32BGUMl63ZBuuhUk3rRSx75n4wnXwn7nYa5VrOwjC19sW\nCF9b34T8MyT8pw6mgauaw9im/xT9sf10Buzm42sm1Fj5Doz8B937fjbmCBvfD9w278v2/wcJ\nSzOfe0Xsp3B90nuEQNILcR22v97RzlL4hC926oyZVm5EZ2zcayKDkhLwv0V8aToKEou9EXZ+\nGMXKD2EMx/wx4piUgNfAd++IP01M8PwwC+PzgyUsPOoL/ah/LHR0+psiY/gByn92fOGRSCz2\n8d6p0p6S4PuWiG+RJr4eYmt+bWRSHr3FfE+K+Jbxul0dWU9aAi5q33P/xrh8AfaJETY0HQxh\nMqtXko6A+X7YI2Ewj8hjZ3iuSvCnmWP4mvxbyL9AdocklZ+hI9zeGPdGP/eK2k9J2yN7lxNI\neiFeUWe7x6E/duT4v7DzDZNWXozO8E1CPZZokhLwSxIm2C0h9v8k+NO8JDKG2xaWpAS8LHQM\n9FXQw+3tC3yovjLix3FHszOhcD/EEvwLA//x0JmYw3XEjjzOj/ithW00pMzC19EfIOGaY6/N\neRE/cg+P9Mp63a6OrCeWCOA2UF6Jv+F2Um92349BjPUJsTfDvhzCI9yXQ8J/2mBKLUkJ+Oup\no7bfRRFu63/VGVOvm9vJf8xiZ0G+Fxnc6OdeUfspssTtJn43oNJ7BFbU2WT+Rxr7L/kHsPPN\nlVZ+hU6eKtojcEr7sAlcB95ooY36hpgRttsS7DQ/FekbGbElmfimTytMfGGJJbNDQ6eKzvdg\nUh9dHoHsz4YrPGX4G6fzQ/gbkOnOxuY0yEchfp/RFhaO5Qd2mYVrWgzhP2G+vBXKsyC235g4\nYt8h8kh3K8SXoX7d+rX4dtL+bXbfb8IkfL+d4CertBmbCYXCwtfIdRC+hykPQxopv6szaF2k\nf1zElmbi0Tv/CT/eCV8TsZLnvbwiFsDZitpPboraphJwLY9e0R6os6GTEvrvTbCHZvo1k4D5\nvVCs8FRzrDwZM1Zs/OBppvy5zuCkfwrCYTyiiJXfxox1bLFYl2JMmIAPhI1nJH5ZiTcJ9Qsr\nbV991Ssltpdhrk9Dxrg5eTT/dsglFdubUIenJftdf8VtoJrkFdcu63XrpqxpxvYXHVqx77nP\nb4LsxYAphVzfWBG6XQM5H1IvodLXl7T3Gv1i79HwTIWPZ+2XofEeyAmQwyH1zrTBZaBssUaG\nut7nXpH7Kbq8LGCiA2XsaAI8Qk0rSd8fPZg2yPXF/JJe3G7YQJMJLTyyMR8eNcXK0zFjxZY0\nJmVITVe9BB77wKkJUFGmxowN2mJnJ3gkdFck3jud7RTXtibHpJ3CN78i6scQ9LuRwO93Nt82\n8w/R4JmBsAzl6zZci9eL3Pf3YKLXQZ7wE2ZonwQf/gMQ+5oibXjSP8c2Ju+ZFP5z9XPIjZCZ\nkCMgWZMvXBM/K9gXlnqfe0Xup3AtA7oScBRL1xvTEhY3fk0CgT0S7KE59t/4/aFTgp73Dcww\nPBVXVKmXgLMm+GdauEBe/BUrl0aMPJq0DzS2w3J5aChZXxSZ7+Ww7QvhhYKvjfQvjthoGsrX\nbcKSBsxF73temzEFcgGE36tnLfz8/3vIaVkHwC/PEWe9sAfCgUfv3N+xwn/GmZgvhPAfhh9B\nwpL0z3roR73e517R+2nQmkYNssjQCwT4wk4rf0zo5Js8S5kccbotYusEU543eNr2rIp0cj+8\nMmKvZ0r6T/5yDPwcZKQLsD/aL4XwH6DjnZ1N/vPwtcBWtno9JrwbcqibmP8w8Ltg8hnt7Gw+\nAPlxYDO1XV+3Zex7nhr+dEVegPpkCP954T4fA0krPApeluZQQB/38TUQnm72hQn+Mghfl/zq\nxH9WfRB6WPL8Q+BjhXGol7GfauZVAq7B0TNKvRftgyDBD2e+SXx5sVcS2gfBzg/9sHRqAg63\no1F9dWTgONgerkikO7eJp2WZnN4QjHwH9Hsh4f5cDhsT2lAXHgWHp0LfBlvs7MMlsG9NWPBQ\nvm55NJlUytj3fm6eWqZ8FsLX2Isgr4DwdRF7D/NMwyTIfZCyyomYKEy+nHsW5P9v5+xCPhvi\nOJ7NJq+RhFoeNmxCrewFstkL9o5ypbS2XGgl8hYX2k1tUuyF9kIpixZRQoRQspFIbtjybu16\njVXE5m2tl+9Xz6nZ386cc579P/N//ut8fvXtzMyZM/M7n5kz8z9zzvmXVjgOdIZgpb4Qsv0X\n7Rr3xt1Oe7zGn3OatP8fga6O6F/Tr2ZO+yKlnZNJT5PWKhIHeu/fnGYaYLi0NLishcVJ2neb\n5KXjU6U+P5gfVL5oPv7SmKj4Q5m0uUjaqEr/CBUvU3x5SHO/vT+kpdFx9dvc9dPWNrXafr5O\nfpF0sXSztEFaI6Xmu77XpNslX7ve5mxhLrFimu/So7kP5Ppvk++EJpBsc2NNsnu3YK7d0gy1\n2imtg/CACKzQuf6T0VQPBldkjnNZfmtyQeF4D/Lu5LFOH+PBItqdSoh5f46ZQjzmd3x1yJNG\nPeDEY3J3VqXvgC9JC8uEN2XKfzmTb57SPsjk/VxppWe694T8HqC8knCiVDIvN34vxXOOcT/v\nOrRUyBykP9LD52d7+DWOfptrxzbfarT9fmLxYYaZr78zWzj5Go19wfGjwzEnF/J5NaXNNmln\nLD93PdyRyee72SMKhV+Zye96ci/xrSjknSqU3STXaKembLYDJLC3HdGoPDh/JcWLyXE/g7xK\nOk06XFomedkol3eX0pdIORvSBOzzXynlGL2h9IXOMG1ealsleUCK+V+ZztO2WZ85LpbjCW+S\nbKmciT7GuFdgumwc/fZtORF9+1VpvhP1NeG2891majXa/hpVEP1w3P3Jy8rRPLn5ha14zJaY\nUfHaE/BlGT/s132Sf1yk5r7hH4zRb8f9yCXaKONejXaK/hEfCIFROqIRnSf9KeU6ft+0u1xQ\nwYY2Ae8vDp9JOXY7le7nds9Lflszl8dpfqGqyxYrQ+n4Jn15VyFzsP/9Fr/9YzB9uazNvdr9\n9okWPxu+q4ODNdr+YNXxbcGXHUp/XFon3S09KZX61Q3aF632BHyKKmxYxe2b2nertEbyBBv3\np3GvCEUbZdyr0U7RP+IDITBKR2wQXa3ALint9H3DT+s4382VbGgTsDmcJflZZV+Gab4bXUBP\n82SeHpuGv9a+vpNZz+pmJdt1LT6vnWENNftt6e4tZXxvxt8abX+66tnewi31KRd+LOOnk2pP\nwK7DjHI+ldJ+zOT/TWle9Uht1HGvRjul/hEeCIFRO2KD6VwFPpVKF0ZM94XipZwuG+IEbCZn\nS6U74cjS8b+ka6WZmPPnynKa74om0bxE6qXc6LfPf2ovHK7Vbw+SL99k/Ez9fq7gb42297L3\n1g5/Ut8cNtOHJd9F52wcE7A5viNF32L8b+Xx+xB+Tu0JN+6/XGmpzca4V6OdUh8JD4DAbHTE\nBpMvVD/bel2KF0AT/0L7HpVynyEpeQ8b6gRsEF4ZWCN9LDX84taDje8SPBjO1I7UAX5pK5bp\n+BkzLWyM+TdmfH5hhPpr9Fu748ngJSnH149tHpBKVqPtvaLhF6S8fJvzqUnzxPuU5Em7zcYx\nAbv++dIt0g6p8bHZ2ldP0BdKjT2jQLO/2b7Y7Jzezta4V6OddnM1PuzebScRCBQIeIL1M5zj\npUOk96TN0g8SNnMCi3SI5UHddwVfStskrzp0vRWuLFlzWVslDyKpvavI4jRhQOEa/faYaZ5u\nv58kP5N9S/IqUB+r0fZHqeLjEv2u8LZp+UfyTmnSzP3eLDyu+Mej+6kn31+kSbAa7TQJ54UP\nEIBABQLrVGZzh5Bub6pQF0VCAAIQgAAEICACvgPKvfG6S+nHQggCEIAABCAAgdEJ+HvPBZI/\nnzhMukD6SErvepuwP5/BIACBCSHgixaDAAT2XQLny3W/VOM3Red1nIaXpbtsqTL4T1ZqmN+6\njS/M1KiHMiEAAQhAAALVCSxRDc0dbtu27a3c1MmVPctrq6u07/q0IsIQGDqBrl/MQ+fD+UNg\n0gn4jeku8zepq7oysR8CEBgvAZagx8ub2iAw2wS2q0B/73tApuBPlLZe2iD529Q+5k9WvuuT\ncS/y+E82MAhAYJoA3wHTFSCw7xOY0in4u09/5+rvfn1X7O89t0h+NoxBAAIQgAAEIAABCEAA\nAhCAAAQgAAEIQAACEIAABCAAAQhAAAIQgAAEIAABCEAAAhCAAAQgAAEIQAACEIAABCAAAQhA\nAAIQgAAEIAABCEAAAhCAAAQgAAEIQAACEIAABCAAAQhAAAIQgAAEIAABCEAAAhCAAAQgAAEI\nQAACEIAABCAAAQhAAAIQgAAEIAABCEAAAhCAAAQgAAEIQAACEIAABCAAAQhAAAIQgAAEIAAB\nCEAAAhCAAAQgAAEIQAACEIAABCAAAQhAAAIQgAAEIAABCEAAAhCAAAQgAAEIQAACEIAABCBQ\nncC/5e8tcQy0lI8AAAAASUVORK5CYII=",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "source('./src/plot_function.R')\n",
    "d[ , plot(romney_voteshare, trump_voteshare, pch = \".\", \n",
    "          xlim=c(0,1), ylim=c(0,1))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit Models on 2012 Returns \n",
    "One of the blocking features that we used in the design of this experiment was the 2012 two-party vote share. As a result, we have very good, orthagonal variation on the percent of the elctoral that voted for a republican vs. democratic candidate and the treatment conditions that we asisgned to each local election offical. There, is, however, no evidence to suggest that differences in the 2012 two-party vote share change the responsiveness of local election officials to our stimulus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "========================================================================\n",
      "                                         Dependent variable:            \n",
      "                              ------------------------------------------\n",
      "                                             GotResponse                \n",
      "                                        OLS                  felm       \n",
      "                                        (1)                   (2)       \n",
      "------------------------------------------------------------------------\n",
      "Latino Cue                             -0.084               -0.066      \n",
      "                                      (0.070)               (0.067)     \n",
      "                                                                        \n",
      "Black Cue                              0.078                 0.088      \n",
      "                                      (0.070)               (0.067)     \n",
      "                                                                        \n",
      "Arab Cue                               -0.018               -0.011      \n",
      "                                      (0.070)               (0.067)     \n",
      "                                                                        \n",
      "Romney Voteshare                      0.215**               -0.057      \n",
      "                                      (0.091)               (0.173)     \n",
      "                                                                        \n",
      "Latino Cue * Romney Voteshare          0.102                 0.068      \n",
      "                                      (0.128)               (0.123)     \n",
      "                                                                        \n",
      "Black Cue * Romney Voteshare           -0.144               -0.163      \n",
      "                                      (0.128)               (0.123)     \n",
      "                                                                        \n",
      "Arab Cue * Romney Voteshare            -0.178               -0.191      \n",
      "                                      (0.127)               (0.122)     \n",
      "                                                                        \n",
      "Constant                              0.499***                          \n",
      "                                      (0.050)                           \n",
      "                                                                        \n",
      "------------------------------------------------------------------------\n",
      "Observations                           6,439                 6,439      \n",
      "R2                                     0.012                 0.338      \n",
      "Adjusted R2                            0.011                 0.113      \n",
      "Residual Std. Error              0.491 (df = 6431)     0.465 (df = 4809)\n",
      "F Statistic                   10.900*** (df = 7; 6431)                  \n",
      "========================================================================\n",
      "Note:                                        *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "m0  = d[ , lm(GotResponse   ~ factor(ethnic_cue) * romney_voteshare)]\n",
    "m0a = d[ , felm(GotResponse ~ factor(ethnic_cue) * romney_voteshare | factor(blockID))]\n",
    "stargazer(m0, m0a, type = \"text\", \n",
    "         covariate.labels = c(\n",
    "         \"Latino Cue\", \"Black Cue\", \"Arab Cue\", \"Romney Voteshare\", \n",
    "         \"Latino Cue * Romney Voteshare\", \"Black Cue * Romney Voteshare\", \"Arab Cue * Romney Voteshare\")\n",
    "         ) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit Models on 2016 Returns \n",
    "We turn to examine whether the relationship between our stimulus and the outcome is mediated by the 2016 electoral outcomes. Here, we fit models that interact the `ethic_cue` treatment variable with Republican two-party voteshare in the election (called `trump_voteshare`). We find that, in models that *do* or *do not* include block indicators that there is a clear effect: geograhies that voted at higher rates for the Republican candidate also discriminate against Arab names at higher rates. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "==================================================\n",
      "                       Dependent variable:        \n",
      "                ----------------------------------\n",
      "                           GotResponse            \n",
      "                  OLS     felm     OLS      felm  \n",
      "                  (1)      (2)     (3)      (4)   \n",
      "--------------------------------------------------\n",
      "Latino           -0.084  -0.066   -0.059   -0.054 \n",
      "                (0.070)  (0.067) (0.070)  (0.068) \n",
      "                                                  \n",
      "Black            0.078    0.088   0.066    0.061  \n",
      "                (0.070)  (0.067) (0.070)  (0.067) \n",
      "                                                  \n",
      "Arab             -0.018  -0.011   0.050    0.050  \n",
      "                (0.070)  (0.067) (0.070)  (0.068) \n",
      "                                                  \n",
      "Romney          0.215**  -0.057                   \n",
      "                (0.091)  (0.173)                  \n",
      "                                                  \n",
      "Latino * Romney  0.102    0.068                   \n",
      "                (0.128)  (0.123)                  \n",
      "                                                  \n",
      "Black * Romney   -0.144  -0.163                   \n",
      "                (0.128)  (0.123)                  \n",
      "                                                  \n",
      "Arab * Romney    -0.178  -0.191                   \n",
      "                (0.127)  (0.122)                  \n",
      "                                                  \n",
      "Trump                             0.058   -0.306**\n",
      "                                 (0.081)  (0.136) \n",
      "                                                  \n",
      "Latino * Trump                    0.048    0.041  \n",
      "                                 (0.114)  (0.110) \n",
      "                                                  \n",
      "Black * Trump                     -0.109   -0.099 \n",
      "                                 (0.113)  (0.109) \n",
      "                                                  \n",
      "Arab * Trump                     -0.271** -0.269**\n",
      "                                 (0.114)  (0.110) \n",
      "                                                  \n",
      "Constant        0.499***         0.579***         \n",
      "                (0.050)          (0.050)          \n",
      "                                                  \n",
      "--------------------------------------------------\n",
      "Observations     6,439    6,439   6,438    6,438  \n",
      "R2               0.012    0.338   0.010    0.340  \n",
      "Adjusted R2      0.011    0.113   0.009    0.116  \n",
      "==================================================\n",
      "Note:                  *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "m1 = d[ ,   lm(GotResponse ~ factor(ethnic_cue) * romney_voteshare)]\n",
    "m2 = d[ , felm(GotResponse ~ factor(ethnic_cue) * romney_voteshare | factor(blockID))]\n",
    "m3 = d[ ,   lm(GotResponse ~ factor(ethnic_cue) * trump_voteshare)]\n",
    "m4 = d[ , felm(GotResponse ~ factor(ethnic_cue) * trump_voteshare | factor(blockID))]\n",
    "stargazer(m1, m2, m3, m4, \n",
    "          type = 'text', omit.stat = c('ser', 'F'), \n",
    "         covariate.labels = c(\n",
    "         \"Latino\", \"Black\", \"Arab\", \n",
    "             \"Romney\", \"Latino * Romney\", \"Black * Romney\", \"Arab * Romney\", \n",
    "             \"Trump\",  \"Latino * Trump\",  \"Black * Trump\",  \"Arab * Trump\")\n",
    "         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = capture.output( \n",
    "    stargazer(m1, m2, m3, m4, \n",
    "              type = 'latex', \n",
    "              out = './figures-tables/trumpHTEtable.tex', \n",
    "              label = 'tab:trumpHTE', \n",
    "              covariate.labels = c('Latino Cue', 'Black Cue', 'Arab Cue', '2012 R. Vote Share', \n",
    "                                   'Latino Cue $\\\\times$ 2012 R. Vote Share',\n",
    "                                   'Black Cue $\\\\times$ 2012 R. Vote Share', \n",
    "                                   'Arab Cue $\\\\times$ 2012 R. Vote Share', \n",
    "                                   '2016 R. Vote Share', \n",
    "                                   'Latino Cue $\\\\times$ 2016 R. Votes Share',\n",
    "                                   'Black Cue $\\\\times$ 2016 R. Vote Share', \n",
    "                                   'Arab Cue $\\\\times$ 2016 R. Vote Share', \n",
    "                                   'Intercept'), \n",
    "              add.lines = list(c('Block Fixed Effects', 'No', 'Yes', 'No', 'Yes')),\n",
    "              omit.stat = c('ser', 'F', 'adj.rsq'), \n",
    "              font.size = \"small\"\n",
    "             )\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>ethnic_cue</th><th scope=col>trump_voteshare</th><th scope=col>fit</th><th scope=col>se.fit</th><th scope=col>low.ci</th><th scope=col>high.ci</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0     </td><td>0.0994</td><td>0.584 </td><td>0.0419</td><td>0.502 </td><td>0.666 </td></tr>\n",
       "\t<tr><td>0     </td><td>0.1082</td><td>0.585 </td><td>0.0412</td><td>0.504 </td><td>0.666 </td></tr>\n",
       "\t<tr><td>0     </td><td>0.1170</td><td>0.585 </td><td>0.0405</td><td>0.506 </td><td>0.665 </td></tr>\n",
       "\t<tr><td>0     </td><td>0.1257</td><td>0.586 </td><td>0.0399</td><td>0.508 </td><td>0.664 </td></tr>\n",
       "\t<tr><td>0     </td><td>0.1345</td><td>0.586 </td><td>0.0392</td><td>0.510 </td><td>0.663 </td></tr>\n",
       "\t<tr><td>0     </td><td>0.1433</td><td>0.587 </td><td>0.0385</td><td>0.511 </td><td>0.662 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " ethnic\\_cue & trump\\_voteshare & fit & se.fit & low.ci & high.ci\\\\\n",
       "\\hline\n",
       "\t 0      & 0.0994 & 0.584  & 0.0419 & 0.502  & 0.666 \\\\\n",
       "\t 0      & 0.1082 & 0.585  & 0.0412 & 0.504  & 0.666 \\\\\n",
       "\t 0      & 0.1170 & 0.585  & 0.0405 & 0.506  & 0.665 \\\\\n",
       "\t 0      & 0.1257 & 0.586  & 0.0399 & 0.508  & 0.664 \\\\\n",
       "\t 0      & 0.1345 & 0.586  & 0.0392 & 0.510  & 0.663 \\\\\n",
       "\t 0      & 0.1433 & 0.587  & 0.0385 & 0.511  & 0.662 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "ethnic_cue | trump_voteshare | fit | se.fit | low.ci | high.ci | \n",
       "|---|---|---|---|---|---|\n",
       "| 0      | 0.0994 | 0.584  | 0.0419 | 0.502  | 0.666  | \n",
       "| 0      | 0.1082 | 0.585  | 0.0412 | 0.504  | 0.666  | \n",
       "| 0      | 0.1170 | 0.585  | 0.0405 | 0.506  | 0.665  | \n",
       "| 0      | 0.1257 | 0.586  | 0.0399 | 0.508  | 0.664  | \n",
       "| 0      | 0.1345 | 0.586  | 0.0392 | 0.510  | 0.663  | \n",
       "| 0      | 0.1433 | 0.587  | 0.0385 | 0.511  | 0.662  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  ethnic_cue trump_voteshare fit   se.fit low.ci high.ci\n",
       "1 0          0.0994          0.584 0.0419 0.502  0.666  \n",
       "2 0          0.1082          0.585 0.0412 0.504  0.666  \n",
       "3 0          0.1170          0.585 0.0405 0.506  0.665  \n",
       "4 0          0.1257          0.586 0.0399 0.508  0.664  \n",
       "5 0          0.1345          0.586 0.0392 0.510  0.663  \n",
       "6 0          0.1433          0.587 0.0385 0.511  0.662  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred.data = data.frame( \n",
    "  ethnic_cue = factor(c(rep(0, 100), rep(1, 100), rep(2, 100), rep(3, 100))), \n",
    "  trump_voteshare = rep(seq(from=min(d$trump_voteshare, na.rm=T), to=max(d$trump_voteshare, na.rm=T), length.out = 100) , 4)\n",
    ")\n",
    "pred.data$fit    = predict.lm(m3, newdata=pred.data, interval = 'confidence')\n",
    "pred.data$se.fit = predict.lm(m3, newdata=pred.data, se.fit = T,)$se.fit\n",
    "pred.data$low.ci = predict.lm(m3, newdata = pred.data, interval = 'confidence', level=0.95)[ , 'lwr']\n",
    "pred.data$high.ci= predict.lm(m3, newdata = pred.data, interval = 'confidence', level=0.95)[ , 'upr']\n",
    "pred.data = data.table(pred.data)\n",
    "head(pred.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "white.col  = rgb(.7, .7, .7, 0.5, 255)\n",
    "latino.col = rgb(0, 0, .7, 0.25)\n",
    "black.col  = rgb(0, 0, 0.3, 0.25)\n",
    "arab.col   = rgb(.7, .7, .7, 0.75)\n",
    "\n",
    "\n",
    "htePlot <- function(alternative=3, title) { \n",
    "    \n",
    "    dens = density(d$trump_voteshare, na.rm=T, from = 0, to=1)\n",
    "    label.vec <- c(\"Latino Cue\", \"Black Cue\", \"Arab Cue\")\n",
    "\n",
    "    source('./src/plot_function.R')\n",
    "    \n",
    "    pred.data[ethnic_cue == 0, \n",
    "          plot(x = trump_voteshare, y = fit, \n",
    "               type = \"n\", ylim = c(0.3, 0.8), col = \"black\", \n",
    "               ylab = \"Probability Respond\",   yaxt = 'n', \n",
    "               xlab = \"Republican Vote Share\", xaxt = 'n',\n",
    "               main = title\n",
    "              )]\n",
    "    axis(1, at = c(seq(0,.8, 0.2), .95), labels = c(\"0\", \"20\", \"40\", \"60\", \"80\", \"100%\"))\n",
    "    axis(2, at = seq(0.2, 0.8, by=0.1), labels = c(\"20\", \"30\", \"40\", \"50\", \"60\", \"70\", \"80%\"))\n",
    "    pred.data[ethnic_cue == 0, \n",
    "          polygon(x = c(trump_voteshare, rev(trump_voteshare)), \n",
    "                  y = c(low.ci, rev(high.ci)), \n",
    "                  border = NA, \n",
    "                  col = white.col)]\n",
    "    pred.data[ethnic_cue == alternative, \n",
    "          polygon(x = c(trump_voteshare, rev(trump_voteshare)), \n",
    "                  y = c(low.ci, rev(high.ci)), \n",
    "                  border = NA, col = arab.col)]\n",
    "    pred.data[ethnic_cue == 0, \n",
    "          points(x = trump_voteshare, y = fit,\n",
    "                 type = \"l\", lty = 1)]\n",
    "    pred.data[ethnic_cue == alternative, \n",
    "          points(x = trump_voteshare, \n",
    "                 y = fit, \n",
    "                 type = \"l\", lty = 2)]\n",
    "    legend(x=.15, y=.45, legend = c(label.vec[alternative], \"White Cue\"), border = NA, \n",
    "#        fill = c(white.col, arab.col), \n",
    "       lty = c(2,1), merge = TRUE,\n",
    "       bty='n', cex = 1.5)\n",
    "\n",
    "    rug(d$trump_voteshare)\n",
    "    points(x = dens$x, y=.3 + dens$y/50, type = 'l') # moving into the plotting window \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pdf(file = './figures-tables/hteTrumpLatino.pdf')\n",
    "htePlot(1, title = \"Latino Cue\")\n",
    "dev.off()\n",
    "\n",
    "pdf(file = './figures-tables/hteTrumpBlack.pdf')\n",
    "htePlot(2, title = \"Black Cue\")\n",
    "dev.off()\n",
    "\n",
    "pdf(file = './figures-tables/hteTrumpArab.pdf')\n",
    "htePlot(3, title = \"Arab/Muslim Cue\")\n",
    "dev.off()\n",
    "\n",
    "pdf(file = './figures-tables/hteTrumpAll.pdf', height=5,width=15)\n",
    "par(mfrow=c(1,3))\n",
    "htePlot(1, title = \"Latino Cue\")\n",
    "htePlot(2, title = \"Black Cue\")\n",
    "htePlot(3, title = \"Arab/Muslim Cue\")\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../figures-tables/hteTrumpArab.pdf'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../figures-tables/hteTrumpAll.pdf' width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models Fit on 2012 v. 2016 Republican Vote Share \n",
    "Is there something unique about areas that moved _uniquely_ toward Trump? That is, given a baseline level of support for the Republican candidate in 2012, did those that moved toward the Republican candidate in 2016 present a unique effect? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "==============================================================\n",
      "                               Dependent variable:            \n",
      "                    ------------------------------------------\n",
      "                                   GotResponse                \n",
      "                              OLS                  felm       \n",
      "                              (1)                   (2)       \n",
      "--------------------------------------------------------------\n",
      "Latino                       -0.009               -0.018      \n",
      "                            (0.027)               (0.026)     \n",
      "                                                              \n",
      "Black                        0.004                -0.011      \n",
      "                            (0.026)               (0.026)     \n",
      "                                                              \n",
      "Arab                         -0.035               -0.042      \n",
      "                            (0.027)               (0.027)     \n",
      "                                                              \n",
      "R Diff                     -0.899***             -0.690**     \n",
      "                            (0.218)               (0.276)     \n",
      "                                                              \n",
      "Romney                      0.220***              -0.060      \n",
      "                            (0.045)               (0.155)     \n",
      "                                                              \n",
      "Latino * R Diff              -0.285               -0.163      \n",
      "                            (0.311)               (0.314)     \n",
      "                                                              \n",
      "Black * R Diff               -0.016                0.201      \n",
      "                            (0.310)               (0.310)     \n",
      "                                                              \n",
      "Arab * R Diff              -1.100***             -1.010***    \n",
      "                            (0.313)               (0.314)     \n",
      "                                                              \n",
      "Constant                    0.554***                          \n",
      "                            (0.030)                           \n",
      "                                                              \n",
      "--------------------------------------------------------------\n",
      "Observations                 6,438                 6,438      \n",
      "R2                           0.032                 0.342      \n",
      "Adjusted R2                  0.031                 0.119      \n",
      "Residual Std. Error    0.486 (df = 6429)     0.464 (df = 4807)\n",
      "F Statistic         26.400*** (df = 8; 6429)                  \n",
      "==============================================================\n",
      "Note:                              *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "m0  = d[ ,   lm(GotResponse ~ factor(ethnic_cue) * I(trump_voteshare - romney_voteshare) +  romney_voteshare)]\n",
    "m1  = d[ , felm(GotResponse ~ factor(ethnic_cue) * I(trump_voteshare - romney_voteshare) +  romney_voteshare | blockID)]\n",
    "stargazer(m0, m1, type = \"text\", \n",
    "    covariate.labels = c(\"Latino\", \"Black\", \"Arab\", \"R Diff\", \"Romney\", \n",
    "                         \"Latino * R Diff\", \"Black * R Diff\", \"Arab * R Diff\")\n",
    "          )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jens Hainmueller, Jonathan Mummolo, and Yiqing Xu suggest a set of clever, [additional checks](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2739221) when interpreting heterogeneous treatment effects via linear interactions. The authors suggest binning the interactive variable into tertiles, and projecting treatment effects at each level of these tertiles. Here, we implement these checks. \n",
    "\n",
    "Because we are most concerned with the role of present campaigning on how the Arab cue was perceived by Local Elections Officials, but because we blocked on past results, the suggetsions of Hainmuller, Mummolo and Xu require some care in construction. We think that the appropriate binning involves movement between 2012 and 2016 Republican vote share. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d[ , delta_rep := trump_voteshare - romney_voteshare] # positive is more republican in 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "p1 <- inter.binning(Y='GotResponse', D='ethnic_cue', X='delta_rep', \n",
    "              data = d[ethnic_cue %in% c(0,1)], \n",
    "              na.rm = T, \n",
    "              Xlabel = 'Delta R Share', \n",
    "              Ylabel = 'Marginal Effect', \n",
    "              main = 'No Latino HTE'      \n",
    "             )$graph\n",
    "p2 <- inter.binning(Y='GotResponse', D='ethnic_cue', X='delta_rep', \n",
    "              data = d[ethnic_cue %in% c(0,2)], \n",
    "              na.rm = T, \n",
    "              Xlabel = 'Delta R Share', \n",
    "              Ylabel = 'Marginal Effect', \n",
    "              main = 'No Black HTE'\n",
    "             )$graph\n",
    "p3 <- inter.binning(Y='GotResponse', D='ethnic_cue', X='delta_rep', \n",
    "              data = d[ethnic_cue %in% c(0,3)], \n",
    "              na.rm = T, \n",
    "              Xlabel = 'Delta R Share', \n",
    "              Ylabel = 'Marginal Effect',\n",
    "              main = 'Strong Arab HTE'\n",
    "             )$graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pdf(file = './figures-tables/hteRobustLatino.pdf')\n",
    "p1\n",
    "dev.off()\n",
    "pdf(file = './figures-tables/hteRobustBlack.pdf')\n",
    "p2\n",
    "dev.off()\n",
    "pdf(file = './figures-tables/hteRobustArab.pdf')\n",
    "p3\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/hteRobustLatino.pdf' width = 700>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/hteRobustBlack.pdf' width = 700>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/hteRobustArab.pdf' width = 800>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also report this for just the `trump_voteshare`; which we do here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p4 <- inter.binning(Y='GotResponse', D='ethnic_cue', X='trump_voteshare', \n",
    "              data = d[ethnic_cue %in% c(0,1)], \n",
    "              na.rm = T, \n",
    "              Xlabel = 'Delta R Share', \n",
    "              Ylabel = 'Marginal Effect', \n",
    "              main = 'No Latino-Trump HTE'      \n",
    "             )$graph\n",
    "p5 <- inter.binning(Y='GotResponse', D='ethnic_cue', X='trump_voteshare', \n",
    "              data = d[ethnic_cue %in% c(0,2)], \n",
    "              na.rm = T, \n",
    "              Xlabel = 'Delta R Share', \n",
    "              Ylabel = 'Marginal Effect', \n",
    "              main = 'No Black-Trump HTE'\n",
    "             )$graph\n",
    "p6 <- inter.binning(Y='GotResponse', D='ethnic_cue', X='trump_voteshare', \n",
    "              data = d[ethnic_cue %in% c(0,3)], \n",
    "              na.rm = T, \n",
    "              Xlabel = 'Delta R Share', \n",
    "              Ylabel = 'Marginal Effect',\n",
    "              main = 'Strong Trump-Arab HTE'      \n",
    "             )$graph\n",
    "\n",
    "pdf('./figures-tables/trump_latino_hte.pdf')\n",
    "p4\n",
    "dev.off()\n",
    "pdf('./figures-tables/trump_black_hte.pdf')\n",
    "p5\n",
    "dev.off()\n",
    "pdf('./figures-tables/trump_arab_hte.pdf')\n",
    "p6\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/trump_latino_hte.pdf' width = 500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/trump_black_hte.pdf' width = 500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/trump_arab_hte.pdf' width = 500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we could also report this on `romney_voteshare`, which we do here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p7 <- inter.binning(Y=\"GotResponse\", D=\"ethnic_cue\", X=\"romney_voteshare\", \n",
    "              data = d[ethnic_cue %in% c(0,1)], \n",
    "              na.rm = T, \n",
    "              Xlabel = \"Delta R Share\", \n",
    "              Ylabel = \"Marginal Effect\", \n",
    "              main = 'No Romney-Latino HTE'\n",
    "             )$graph\n",
    "p8 <- inter.binning(Y=\"GotResponse\", D=\"ethnic_cue\", X=\"romney_voteshare\", \n",
    "              data = d[ethnic_cue %in% c(0,2)], \n",
    "              na.rm = T, \n",
    "              Xlabel = \"Delta R Share\", \n",
    "              Ylabel = \"Marginal Effect\",\n",
    "              main = 'No Romney-Black HTE'\n",
    "             )$graph\n",
    "p9 <- inter.binning(Y=\"GotResponse\", D=\"ethnic_cue\", X=\"romney_voteshare\", \n",
    "              data = d[ethnic_cue %in% c(0,3)], \n",
    "              na.rm = T, \n",
    "              Xlabel = \"Delta R Share\", \n",
    "              Ylabel = \"Marginal Effect\",\n",
    "              main = 'Strong Romney-Arab HTE'\n",
    "             )$graph\n",
    "\n",
    "pdf('./figures-tables/romney_latino_hte.pdf')\n",
    "p7\n",
    "dev.off()\n",
    "\n",
    "pdf('./figures-tables/romney_black_hte.pdf')\n",
    "p8\n",
    "dev.off()\n",
    "\n",
    "pdf('./figures-tables/romney_arab_hte.pdf')\n",
    "p9\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/romney_latino_hte.pdf' width = 500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/romney_black_hte.pdf' width = 500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = '../figures-tables/romney_arab_hte.pdf' width = 500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"vra5\"></a>\n",
    "## 3.C Section 5 VRA Status\n",
    "[toc](#toc)\n",
    "\n",
    "Changes in VRA coverage for some registrars raises the possiblity that some Registrars may respond differentially to treatment stimulus than others. Indeed, as White Nathan and Faller (2015) note: \n",
    "\n",
    "> \"private sector employees are less likely to be discriminatory in discretionary decisions about clients when organizational procedures are in place to make them aware of the risk for discrimination and to explicitly monitor against it (Pager and Shepherd, 2008, 14–7). If similar dynamics hold for election officials, biased behavior should be less likely when institutional mechanisms are in place to prevent it (White, Nathan and Faller 2015, p. 137). \n",
    "\n",
    "If the nature of the bias demonstrated by local election officials in the 2016 election was as a consequence of _implicit_ bias then we might expect a certain pattern to the discriminatory behavior against minority questions. \n",
    "\n",
    "First, whereas White, Nathan and Faller (2015) found suggestive, but not-statistically-significant evidence that areas under requirements of VRA $\\S 5$ coveragse were _less_ biased than those areas that were not under such coverage, if the nature of the bias against minoritites is implicit then we should expect that removing the $\\S 5$ requirements should remove this protective effect -- that is, we should expect there to be no difference in the behavior of previously $\\S 5$-covered vs. non-covered juristictions. \n",
    "\n",
    "Second, in the case of implicit bias we should expect that increasing populations of a minority group in a geography should be associated with a _decreased_ presence of bias against that minority group. A central tenant of \"contact theory\" is that close proximity, and common purpose increases feelings of well-being toward members of out-groups. While the preconditions for social contact to ameoliorate racial animus are more stringent than simple co-location, for the purposes of this study, we presume that co-location proxies for many of these preconditions. In this way, although a lack of conditional bias would be very weak evidence for a lack of implicit bias, the presence of a conditional effect in this model is compelling evidence for the possibility that the drivers of bias are implicit in nature. \n",
    "\n",
    "\n",
    "At the time White, Nathan and Faller conducted their experiment, VRA protections were stil in place in 706 + 186 = 892 jurisdictions. At that time the authors determine that areas covered with VRA protections did *not* demonstrate any pattern of discriminatory behavior toward minority (Latino) constituents, while areas un-covered by VRA protectiosn *did* demonstrate such a pattern. \n",
    "\n",
    "Our experiment finds the same pattern of behavior: there is no pattern of systematic discrimination among Registrars who were previously covered by VRA protections. Indeed, if anything, the registrars in these jurisdictions were more responsive than their counterparts in non-VRA covered jurisdictions. However, we are quick to point out two facts concerning this relationship. First, this is an undepowerd test with fewer than 1,000 observations, meaning fewer than 250 per condition. Second, this test is *essentially* non-experimental in nature, meaning that any of a large number of feature that covary with the conditioning feature may be feature \"responsible\" for the conditional relationship.\n",
    "\n",
    "In the analyis that follows we report the conditional ATE within both VRA and non-VRA covered districts, and provide a formal test for whether behavior is different between these jurisdictions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many counties do we identify as having been previously covered by VRA? <!--**NOTE** why is our tally different than the tally that was produced by White, Nathan and Faller (2015)? --> <!--**ANSWER TO NOTE, which requires follow-up analysis: what we are reporting is _all_ causes of VRA coverage, not just \\S 5. **NK: Can you confirm?** --> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Covered by VRA</th><th scope=col>Total Registrars</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>No  </td><td>5672</td></tr>\n",
       "\t<tr><td>Yes </td><td> 767</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Covered by VRA & Total Registrars\\\\\n",
       "\\hline\n",
       "\t No   & 5672\\\\\n",
       "\t Yes  &  767\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Covered by VRA | Total Registrars | \n",
       "|---|---|\n",
       "| No   | 5672 | \n",
       "| Yes  |  767 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Covered by VRA Total Registrars\n",
       "1 No             5672            \n",
       "2 Yes             767            "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d[ , .(totalRegistrars = .N), keyby = .(\"Covered by VRA\" = vra_county_state)] %>% \n",
    "  .[ , .(\"Covered by VRA\" = c(\"No\", \"Yes\"),\n",
    "         \"Total Registrars\" = totalRegistrars)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The baseline response rate is considerably higher in areas that were $\\S 5$ covered until 2013. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Covered by VRA</th><th scope=col>Total Responses</th><th scope=col>Response Rate</th><th scope=col>Standard Error</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>No     </td><td>3217   </td><td>0.567  </td><td>0.00658</td></tr>\n",
       "\t<tr><td>Yes    </td><td> 505   </td><td>0.658  </td><td>0.01714</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " Covered by VRA & Total Responses & Response Rate & Standard Error\\\\\n",
       "\\hline\n",
       "\t No      & 3217    & 0.567   & 0.00658\\\\\n",
       "\t Yes     &  505    & 0.658   & 0.01714\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Covered by VRA | Total Responses | Response Rate | Standard Error | \n",
       "|---|---|\n",
       "| No      | 3217    | 0.567   | 0.00658 | \n",
       "| Yes     |  505    | 0.658   | 0.01714 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Covered by VRA Total Responses Response Rate Standard Error\n",
       "1 No             3217            0.567         0.00658       \n",
       "2 Yes             505            0.658         0.01714       "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d[ , .(totalRespond   = sum(GotResponse), \n",
    "       percentRespond = sum(GotResponse) / .N, \n",
    "       se = sqrt(var(GotResponse) / .N)), \n",
    "  keyby = .(\"Covered by VRA\" = vra_county_state)] %>% \n",
    "  .[ , .(\"Covered by VRA\" = c(\"No\", \"Yes\"), \n",
    "         \"Total Responses\" = totalRespond, \n",
    "         \"Response Rate\" = percentRespond, \n",
    "         \"Standard Error\" = se) ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This underlies an important non-experimental finding: areas that were previously covered by the VRA were considerably more likely to respond to *any* form of purported constituent contact. Indeed, registrars previously covered by the VRA were 9 percent more likely to respond to a query than registrars not previously covered. Our finding (reported above) is consistent with previous findings that VRA covered Registrars were more likely to respond to queries. This is reported using the White, Nathan and Faller (2015) data, below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>WNF Findings</b>:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Covered by VRA</th><th scope=col>Response Rate</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>No  </td><td>0.65</td></tr>\n",
       "\t<tr><td>Yes </td><td>0.69</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Covered by VRA & Response Rate\\\\\n",
       "\\hline\n",
       "\t No   & 0.65\\\\\n",
       "\t Yes  & 0.69\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Covered by VRA | Response Rate | \n",
       "|---|---|\n",
       "| No   | 0.65 | \n",
       "| Yes  | 0.69 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Covered by VRA Response Rate\n",
       "1 No             0.65         \n",
       "2 Yes            0.69         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_html(\"<b>WNF Findings</b>:\" )\n",
    "rep.d[ , .(responseRate = mean(response, na.rm = TRUE)), \n",
    "      keyby = .(\"Covered by VRA\" = preclear_states)] %>% \n",
    "  .[ , .(\"Covered by VRA\" = c(\"No\", \"Yes\"), \n",
    "         \"Response Rate\" = responseRate) ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to highlight that this is an *essentially* non-experimental finding. That is, the comparison that would provide causal evidence for the performance of the VRA in increasing the responsiveness of registrars would compare VRA areas against themselves. In the comparison that is made here as well as in White, Nathan and Faller (2015), Registrars who are and are not covered are compared to one another, and so clear causal statements about the mechanisms leading to differential behavior are not possible. \n",
    "\n",
    "To examine whether Registrars' response to treatment is different conditional on being a Registrar in a previously VRA covered area vs. an area that was has not been VRA covered we estimate models that interact VRA coverage with our treatment indicator. A significant finding on these interactions would provide evidence that treatment functions differently among Registrars in these distinct areas. We find little evidence of such an effect. \n",
    "\n",
    "### No VRA $\\S 5$ Conditional Effect \n",
    "*There is scant evidence of a conditional effect*. This study does not produce evidence that local election offficls behavior toward minority voters is different in areas that were previously covered by VRA $\\S 5$ than those that were not. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b> F-test for Interactions: Null vs. Minority Model </b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Res.Df</th><th scope=col>RSS</th><th scope=col>Df</th><th scope=col>Sum of Sq</th><th scope=col>F</th><th scope=col>Pr(&gt;F)</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>6435   </td><td>1562   </td><td>NA     </td><td>     NA</td><td>    NA </td><td>  NA   </td></tr>\n",
       "\t<tr><td>6436   </td><td>1562   </td><td>-1     </td><td>-0.0172</td><td>0.0707 </td><td>0.79   </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " Res.Df & RSS & Df & Sum of Sq & F & Pr(>F)\\\\\n",
       "\\hline\n",
       "\t 6435    & 1562    & NA      &      NA &     NA  &   NA   \\\\\n",
       "\t 6436    & 1562    & -1      & -0.0172 & 0.0707  & 0.79   \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Res.Df | RSS | Df | Sum of Sq | F | Pr(>F) | \n",
       "|---|---|\n",
       "| 6435    | 1562    | NA      |      NA |     NA  |   NA    | \n",
       "| 6436    | 1562    | -1      | -0.0172 | 0.0707  | 0.79    | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Res.Df RSS  Df Sum of Sq F      Pr(>F)\n",
       "1 6435   1562 NA      NA       NA   NA  \n",
       "2 6436   1562 -1 -0.0172   0.0707 0.79  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b> F-test for Interactions: Null vs. All Cues</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Res.Df</th><th scope=col>RSS</th><th scope=col>Df</th><th scope=col>Sum of Sq</th><th scope=col>F</th><th scope=col>Pr(&gt;F)</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>6431  </td><td>1551  </td><td>NA    </td><td>    NA</td><td>   NA </td><td>   NA </td></tr>\n",
       "\t<tr><td>6434  </td><td>1551  </td><td>-3    </td><td>-0.544</td><td>0.752 </td><td>0.521 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " Res.Df & RSS & Df & Sum of Sq & F & Pr(>F)\\\\\n",
       "\\hline\n",
       "\t 6431   & 1551   & NA     &     NA &    NA  &    NA \\\\\n",
       "\t 6434   & 1551   & -3     & -0.544 & 0.752  & 0.521 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Res.Df | RSS | Df | Sum of Sq | F | Pr(>F) | \n",
       "|---|---|\n",
       "| 6431   | 1551   | NA     |     NA |    NA  |    NA  | \n",
       "| 6434   | 1551   | -3     | -0.544 | 0.752  | 0.521  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Res.Df RSS  Df Sum of Sq F     Pr(>F)\n",
       "1 6431   1551 NA     NA       NA    NA \n",
       "2 6434   1551 -3 -0.544    0.752 0.521 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hte.mod.minority.null  = d[ , lm(GotResponse ~ I(ethnic_cue != 0) + factor(vra_county_state))]\n",
    "hte.mod.minority       = d[ , lm(GotResponse ~ I(ethnic_cue != 0) * factor(vra_county_state))]\n",
    "hte.mod.cue.null       = d[ , lm(GotResponse ~ factor(ethnic_cue) + factor(vra_county_state))]    \n",
    "hte.mod.cue            = d[ , lm(GotResponse ~ factor(ethnic_cue) * factor(vra_county_state))]    \n",
    "hte.mod.minority.felm  = d[ , felm(GotResponse ~ I(ethnic_cue != 0) * factor(vra_county_state) | factor(blockID))]\n",
    "hte.mod.cue.felm       = d[ , felm(GotResponse ~ factor(ethnic_cue) * factor(vra_county_state) | factor(blockID))]\n",
    "\n",
    "display_html(\"<b> F-test for Interactions: Null vs. Minority Model </b>\")\n",
    "anova(hte.mod.minority, hte.mod.minority.null, test = \"F\")\n",
    "\n",
    "display_html(\"<b> F-test for Interactions: Null vs. All Cues</b>\")\n",
    "anova(hte.mod.cue, hte.mod.cue.null)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=============================================================================================\n",
      "                                                Dependent variable:                          \n",
      "                      -----------------------------------------------------------------------\n",
      "                                                    GotResponse                              \n",
      "                             (1)               (2)               (3)               (4)       \n",
      "---------------------------------------------------------------------------------------------\n",
      "Minority Cue              -0.047***         -0.045***                                        \n",
      "                           (0.014)           (0.015)                                         \n",
      "                                                                                             \n",
      "Latino Cue                                                     -0.029*           -0.033*     \n",
      "                                                               (0.017)           (0.018)     \n",
      "                                                                                             \n",
      "Black Cue                                                       0.001             0.007      \n",
      "                                                               (0.017)           (0.018)     \n",
      "                                                                                             \n",
      "Arab Cue                                                      -0.112***         -0.110***    \n",
      "                                                               (0.017)           (0.018)     \n",
      "                                                                                             \n",
      "VRA County                0.091***          0.100***          0.091***          0.100***     \n",
      "                           (0.019)           (0.038)           (0.019)           (0.038)     \n",
      "                                                                                             \n",
      "Minority * VRA County                        -0.012                                          \n",
      "                                             (0.044)                                         \n",
      "                                                                                             \n",
      "Latino * VRA County                                                               0.030      \n",
      "                                                                                 (0.053)     \n",
      "                                                                                             \n",
      "Black * VRA County                                                               -0.048      \n",
      "                                                                                 (0.053)     \n",
      "                                                                                             \n",
      "Arab * VRA County                                                                -0.017      \n",
      "                                                                                 (0.053)     \n",
      "                                                                                             \n",
      "Constant                  0.602***          0.601***          0.602***          0.601***     \n",
      "                           (0.012)           (0.013)           (0.012)           (0.013)     \n",
      "                                                                                             \n",
      "---------------------------------------------------------------------------------------------\n",
      "Observations                6,439             6,439             6,439             6,439      \n",
      "R2                          0.005             0.005             0.012             0.013      \n",
      "Adjusted R2                 0.005             0.005             0.012             0.012      \n",
      "Residual Std. Error   0.493 (df = 6436) 0.493 (df = 6435) 0.491 (df = 6434) 0.491 (df = 6431)\n",
      "=============================================================================================\n",
      "Note:                                                             *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "stargazer(hte.mod.minority.null, hte.mod.minority, hte.mod.cue.null, hte.mod.cue, \n",
    "         type = \"text\",\n",
    "         covariate.labels = c(\"Minority Cue\", \"Latino Cue\", \"Black Cue\", \"Arab Cue\", \n",
    "                              \"VRA County\", \"Minority * VRA County\", \n",
    "                              \"Latino * VRA County\", \"Black * VRA County\", \n",
    "                              \"Arab * VRA County\"), \n",
    "          omit.stat = \"F\"\n",
    "         )\n",
    "\n",
    "m = capture.output(\n",
    "    stargazer(hte.mod.minority.null, hte.mod.minority, hte.mod.cue.null, hte.mod.cue, \n",
    "              type = \"latex\", \n",
    "              out = \"./figures-tables/VRAmodels.tex\",\n",
    "              covariate.labels = c(\"Minority Cue\", \"Latino Cue\", \"Black Cue\", \"Arab Cue\", \n",
    "                              \"VRA County\", \"Minority * VRA County\", \n",
    "                              \"Latino * VRA County\", \"Black * VRA County\", \n",
    "                              \"Arab * VRA County\"), \n",
    "              title = \"HTE of Prior VRA Status\", \n",
    "              label = \"tab:vraTable\"\n",
    "              )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the differential hypothesis that we would expect to see if VRA removal had led to differential behavior? That in districts that _used_ to be VRA covered, but are now not, _and_ that broke Republican in the 2016 election had differential bias against minorities than places that broke Republican but were not previously covered by the VRA. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. HIT Tracker Work \n",
    "<a id = \"hit\"></a>\n",
    "\n",
    "In our work, a primary contribution that has not been previously made is to include a measurement for whether a recipient _opened_ the emails that they were sent. However, building the measurments for this process requires some care which we present here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Tracked \n",
    "In the file `./replication_tracker.csv` we have a record for _every_ server hit, along with the associated registrar ID. Each entry in this file is a time stamp of (a) when our server received a request to load the tracking pixel, and (b) the associated registrar for that request. \n",
    "\n",
    "Due to the nature of modern email, it is possible, and quite likely that there will not be a one-to-one mapping between a tracker hit, and an email open by the local election official. Here we list some of the reasons for these possiblities: \n",
    "- Some individuals' email clients set a flag to that keeps them from loading images, including ours. This is a relatively uncommon event, but it is the most common among individuals who use Microsoft Office email clinets. \n",
    "- Some individuals' email clients load a `cached` version of an image from a proxy server. When the client observes that there is an image, it downloads that image to their own servers and checks these images for viruses, etc. Subsequent emails that are opened, and detect the same image, load from the proxy server, rather than our server. In our data, this event occurred when registrars utilized gmail email services. \n",
    "\n",
    "In the analysis for this section, we take the following steps: \n",
    "- We calculate a variable we call `gmail_proxy` that records whether *at any point* we received a correspondence from a local election official via a gmail server. \n",
    "- We count the time to open, and the number of opens that we observe from an individual\n",
    "- Omit email addresses that we sent to that contain an `@gmail.com` top-level domain, aware that there may be individuals who are using Gmail or other advanced clients through email forwarding or other methods. \n",
    "\n",
    "Finally, we estimate the following models: \n",
    "- Models that demonstrate that the core results of the paper continue to exist even when omitting local elections officials who use `@gmail.com` email addresses\n",
    "- How the estimated models change if we restrict the analysis to local election officials who **actually open** the emails that we send them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tracker = fread('./data/replication_tracker.csv')\n",
    "tracker[ , timestamp_d := ymd_hms(timestamp)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "short_tracker <- tracker %>% \n",
    "  .[ , .(gmail_proxy = sum(gmail_proxy, na.rm = T) > 0 , \n",
    "         first_hit   = min(timestamp_d, na.rm = T), \n",
    "         last_hit    = max(timestamp_d, na.rm = T),\n",
    "         n_hits      = .N), \n",
    "    keyby = .(registrar_id)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in `tracker` we had multiple observations (each time the tracker loaded) for a subset of the LEO universe that was under consdieration -- those who loaded the webpage. \n",
    "\n",
    "In `short_tracker` we have a _single_ observation -- that reports the first and last time the server was hit by a LEO -- for the same subset of the LEO universe that loaded the webpage. \n",
    "\n",
    "In the next cell, we're going to merge with the core data table, keeping all observations from that data table -- the entire experimental population of our LEO. Then, we convert some variables to date format, calculate time between sending a first tracker hit. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "TRUE"
      ],
      "text/latex": [
       "TRUE"
      ],
      "text/markdown": [
       "TRUE"
      ],
      "text/plain": [
       "[1] TRUE"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tracker = merge(\n",
    "    d[ , .(ethnic_cue, GotResponse, blockID, registrar_id, gmail_address,\n",
    "           RecvDate, SentDate, HitTracker, trump_voteshare, \n",
    "           vra_county_state, dens_log, pct_inc2pov_150, pct_race_black, pct_latino, \n",
    "           obama_margin, state)], \n",
    "    short_tracker, by.x = 'registrar_id', by.y = 'registrar_id', \n",
    "    all.x = T)\n",
    "d[ , .N] == tracker[ , .N]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Throughout the rest of this analysis of opening an email, we're going to omit those registrars who have `@gmail.com` email clients. Randomization ensures that there are equal proportions of these registrars in each experimental cell. We pull this from the email address that we sent to, rather than the proxy load. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>ethnic_cue</th><th scope=col>gmail proxy loads</th><th scope=col>gmail addresses</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0  </td><td>147</td><td>96 </td></tr>\n",
       "\t<tr><td>1  </td><td>126</td><td>86 </td></tr>\n",
       "\t<tr><td>2  </td><td>145</td><td>80 </td></tr>\n",
       "\t<tr><td>3  </td><td>130</td><td>82 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " ethnic\\_cue & gmail proxy loads & gmail addresses\\\\\n",
       "\\hline\n",
       "\t 0   & 147 & 96 \\\\\n",
       "\t 1   & 126 & 86 \\\\\n",
       "\t 2   & 145 & 80 \\\\\n",
       "\t 3   & 130 & 82 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "ethnic_cue | gmail proxy loads | gmail addresses | \n",
       "|---|---|---|---|\n",
       "| 0   | 147 | 96  | \n",
       "| 1   | 126 | 86  | \n",
       "| 2   | 145 | 80  | \n",
       "| 3   | 130 | 82  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  ethnic_cue gmail proxy loads gmail addresses\n",
       "1 0          147               96             \n",
       "2 1          126               86             \n",
       "3 2          145               80             \n",
       "4 3          130               82             "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tracker[is.na(gmail_proxy), gmail_proxy := FALSE] # this fixes the missingness here. \n",
    "tracker[ , .('gmail proxy loads' = sum(gmail_proxy, na.rm=TRUE), \n",
    "             'gmail addresses'   = sum(gmail_address, na.rm = TRUE)), \n",
    "        keyby = ethnic_cue]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tracker[ , ':='(\n",
    "    RecvDate_d = ymd_hms(RecvDate), \n",
    "    SentDate_d = ymd_hms(SentDate))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker = tracker[ , ':='(\n",
    "  first_seconds_d = first_hit - SentDate_d, \n",
    "  last_seconds_d  = last_hit  - SentDate_d)]\n",
    "tracker = tracker[ , ':='(\n",
    "  first_seconds_n = as.numeric(first_seconds_d), \n",
    "  last_seconds_n  = as.numeric(last_seconds_d))]\n",
    "tracker = tracker[ , ':='(\n",
    "    first_minutes_n = first_seconds_n / 60, \n",
    "    last_minutes_n  = last_seconds_n  / 60)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, and critically, we build an indicator for whether we received a tracker hit from a Local Election Official. This is just drawn as a boolean indicator of whether the `first_hit` variable was `NA`. If it was `NA`, then we did not receive a tracker hit. If it was not `NA` we say that we received a tracker hit. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker = tracker[ , ':='(\n",
    "    tracker_hit = !(is.na(first_hit)))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cross-tab of tracker hits and responses is reported below, omitting the registrars that use gmail addresses. The results of this table seem quite reasonable to the authors. On the one hand, 2,082 individuals that we did *not* receive a response from did *not* send us a tracker reply; similarily 2,089 individuals that we *did* receive a reply from also registered a tracker hit. \n",
    "\n",
    "The limitations of the `F1` score in measuring open rates is evident. The F1 score we generate is 0.68, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "f1 <- function(tab) { \n",
    "  precision <- diag(tab) / rowSums(tab) # Precision = TP/TP+FP\n",
    "  recall    <- diag(tab) / colSums(tab) # Recall    = TP/TP+FN\n",
    "  f1 <- (2 * precision * recall) / (precision + recall)\n",
    "      sprintf(\"Precision: %.2f, Recall: %.2f, F1: %.2f\", \n",
    "         precision[2], recall[2], f1[2])\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b> Gmail Address: </b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "           Received Response\n",
       "Tracker Hit FALSE TRUE\n",
       "      FALSE  2082 1385\n",
       "      TRUE    539 2089"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'Precision: 0.79, Recall: 0.60, F1: 0.68'"
      ],
      "text/latex": [
       "'Precision: 0.79, Recall: 0.60, F1: 0.68'"
      ],
      "text/markdown": [
       "'Precision: 0.79, Recall: 0.60, F1: 0.68'"
      ],
      "text/plain": [
       "[1] \"Precision: 0.79, Recall: 0.60, F1: 0.68\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_html(\"<b> Gmail Address: </b>\")\n",
    "t1 = tracker[gmail_address==FALSE , table('Tracker Hit' = tracker_hit, \n",
    "                                     'Received Response' = GotResponse)]\n",
    "t1 \n",
    "f1(t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b> Gmail Proxy: </b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "           Received Response\n",
       "Tracker Hit FALSE TRUE\n",
       "      FALSE  2125 1404\n",
       "      TRUE    467 1895"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'Precision: 0.80, Recall: 0.57, F1: 0.67'"
      ],
      "text/latex": [
       "'Precision: 0.80, Recall: 0.57, F1: 0.67'"
      ],
      "text/markdown": [
       "'Precision: 0.80, Recall: 0.57, F1: 0.67'"
      ],
      "text/plain": [
       "[1] \"Precision: 0.80, Recall: 0.57, F1: 0.67\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_html(\"<b> Gmail Proxy: </b>\")\n",
    "t2 = tracker[gmail_proxy==FALSE , table('Tracker Hit' = tracker_hit, \n",
    "                                     'Received Response' = GotResponse)]\n",
    "t2\n",
    "f1(t2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>ethnic_cue</th><th scope=col>tracker_hits</th><th scope=col>responses</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0  </td><td>679</td><td>917</td></tr>\n",
       "\t<tr><td>1  </td><td>682</td><td>883</td></tr>\n",
       "\t<tr><td>2  </td><td>679</td><td>928</td></tr>\n",
       "\t<tr><td>3  </td><td>588</td><td>746</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " ethnic\\_cue & tracker\\_hits & responses\\\\\n",
       "\\hline\n",
       "\t 0   & 679 & 917\\\\\n",
       "\t 1   & 682 & 883\\\\\n",
       "\t 2   & 679 & 928\\\\\n",
       "\t 3   & 588 & 746\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "ethnic_cue | tracker_hits | responses | \n",
       "|---|---|---|---|\n",
       "| 0   | 679 | 917 | \n",
       "| 1   | 682 | 883 | \n",
       "| 2   | 679 | 928 | \n",
       "| 3   | 588 | 746 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  ethnic_cue tracker_hits responses\n",
       "1 0          679          917      \n",
       "2 1          682          883      \n",
       "3 2          679          928      \n",
       "4 3          588          746      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tracker[gmail_address==FALSE] %>% \n",
    "  .[, .(tracker_hits = sum(tracker_hit, na.rm = TRUE), \n",
    "        responses    = sum(GotResponse, na.rm = TRUE)), \n",
    "    keyby = ethnic_cue]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "0.45193353005125"
      ],
      "text/latex": [
       "0.45193353005125"
      ],
      "text/markdown": [
       "0.45193353005125"
      ],
      "text/plain": [
       "[1] 0.452"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tracker[ , mean(tracker_hit)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "mod0 = tracker[gmail_address==FALSE, felm(tracker_hit ~ 1 | blockID)]\n",
    "mod1 = tracker[gmail_address==FALSE, felm(tracker_hit ~ 1 + factor(ethnic_cue != 0) | blockID)]\n",
    "mod2 = tracker[gmail_address==FALSE, felm(tracker_hit ~ 1 + factor(ethnic_cue) | blockID)]\n",
    "mod3 = tracker[gmail_address==FALSE & tracker_hit == TRUE, \n",
    "               felm(GotResponse ~ 1 + factor(ethnic_cue != 0) | blockID)]\n",
    "mod4 = tracker[gmail_address==FALSE & tracker_hit == TRUE, \n",
    "               felm(GotResponse ~ 1 + factor(ethnic_cue) | blockID)]\n",
    "mod5 = tracker[gmail_address==FALSE & tracker_hit == TRUE, \n",
    "              lm(GotResponse ~ factor(ethnic_cue))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=================================================\n",
      "                     Dependent variable:         \n",
      "             ------------------------------------\n",
      "                TrackerHit        GotResponse    \n",
      "               (1)      (2)      (3)      (4)    \n",
      "-------------------------------------------------\n",
      "Minority Cue -2.590*           -3.530            \n",
      "             (1.450)           (2.350)           \n",
      "                                                 \n",
      "Latino Cue            -0.235             -0.289  \n",
      "                      (1.770)           (2.830)  \n",
      "                                                 \n",
      "Black Cue             -0.742             -0.907  \n",
      "                      (1.760)           (2.790)  \n",
      "                                                 \n",
      "Arab Cue             -6.790***         -10.100***\n",
      "                      (1.770)           (3.110)  \n",
      "                                                 \n",
      "-------------------------------------------------\n",
      "Observations  6,095    6,095    2,628    2,628   \n",
      "R2            0.303    0.305    0.572    0.577   \n",
      "=================================================\n",
      "Note:                 *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "stargazer(mod1, mod2, mod3, mod4, \n",
    "          type = 'text',\n",
    "          covariate.labels = c(\"Minority Cue\", \"Latino Cue\", \"Black Cue\", \"Arab Cue\"),\n",
    "          apply.coef = function(x) x * 100, \n",
    "          apply.se = function(x) x * 100,\n",
    "          se = list(felm_rses(mod1), felm_rses(mod2), felm_rses(mod3), felm_rses(mod4)),\n",
    "          omit.stat = c(\"ser\", \"adj.rsq\"), \n",
    "#           notes = \"Regression estimates with White Standard errors. \\\\textit{TrackerHit} is a measured open, and \\\\textit{GotResponse} is receiving a response from an election official. All models include block fixed-effects. Coefficients are reported as percentage points.\", \n",
    "          dep.var.labels = c(\"TrackerHit\", \"GotResponse\"), \n",
    "          single.row = FALSE, \n",
    "          header = FALSE)   \n",
    "\n",
    "# summary(mod4)\n",
    "m = capture.output(\n",
    "    stargazer(mod1, mod2, mod3, mod4, \n",
    "          type = 'latex', \n",
    "          out = \"./figures-tables/trackerHitModel.tex\", \n",
    "          covariate.labels = c(\"Minority Cue\", \"Latino Cue\", \"Black Cue\", \"Arab Cue\"),\n",
    "          apply.coef = function(x) x * 100, \n",
    "          apply.se = function(x) x * 100,\n",
    "          se = list(felm_rses(mod1), felm_rses(mod2), felm_rses(mod3), felm_rses(mod4)),\n",
    "          omit.stat = c(\"ser\", \"adj.rsq\"), \n",
    "          notes = \"Regression estimates with White Standard errors. \\\\textit{TrackerHit} is a measured open, and \\\\textit{GotResponse} is receiving a response from an election official. All models include block fixed-effects. Coefficients are reported as percentage points.\", \n",
    "          dep.var.labels = c(\"TrackerHit\", \"GotResponse\"), \n",
    "          single.row = FALSE, \n",
    "          header = FALSE)          \n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gmail_address\n",
       "FALSE  TRUE \n",
       " 6095   344 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tracker[ , table(gmail_address)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================================================================\n",
      "                                            Dependent variable:                    \n",
      "                        -----------------------------------------------------------\n",
      "                                                GotResponse                        \n",
      "                                       felm                           OLS          \n",
      "                               (1)               (2)                  (3)          \n",
      "-----------------------------------------------------------------------------------\n",
      "factor(ethnic_cue != 0)      -0.035*                                               \n",
      "                             (0.021)                                               \n",
      "                                                                                   \n",
      "factor(ethnic_cue)1                            -0.003               -0.001         \n",
      "                                               (0.026)              (0.022)        \n",
      "                                                                                   \n",
      "factor(ethnic_cue)2                            -0.009               -0.010         \n",
      "                                               (0.026)              (0.022)        \n",
      "                                                                                   \n",
      "factor(ethnic_cue)3                           -0.101***            -0.094***       \n",
      "                                               (0.027)              (0.023)        \n",
      "                                                                                   \n",
      "Constant                                                           0.819***        \n",
      "                                                                    (0.015)        \n",
      "                                                                                   \n",
      "-----------------------------------------------------------------------------------\n",
      "Observations                  2,628             2,628                2,628         \n",
      "R2                            0.572             0.577                0.009         \n",
      "Adjusted R2                   0.111             0.121                0.008         \n",
      "Residual Std. Error     0.381 (df = 1266) 0.379 (df = 1264)    0.402 (df = 2624)   \n",
      "F Statistic                                                 7.830*** (df = 3; 2624)\n",
      "===================================================================================\n",
      "Note:                                                   *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "stargazer(mod3, mod4, mod5, type = 'text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"interruption\"></a> \n",
    "# 5. Interruption by Notification\n",
    "[toc](#toc)\n",
    "\n",
    "As we note in the paper, on November 2nd, an email was sent from the NASS, the professional organization for secrataaries of state. While this event presents a potential problem of interference between units, the responses collected from units *before* the interference event can reasonably be used to draw inference. \n",
    "\n",
    "However, at the same time, the nature of the data generating procedure -- an audit study where the failure to obtain a response from an experiemntal unit is a meaningful piece of data -- may raise some questions about how the notification of secretaries of state affected the results of this audit study. We make three claimss. \n",
    "\n",
    "1. In this study, the time at which widespread notification was sent out left little data to *potentially* effect causal effect estimates. \n",
    "2. Responses collected before widespread notification are meaningful data points; \n",
    "3. Causal estimates will only be biased if notification causes conditionally different response to treatment. \n",
    "\n",
    "First and foremost, in this study as well as the study conduced by White, Nathan, and Faller (2015) as well as several other audit studies (e.g. Gell-Redman (*in press*), Crabtree (*in press*)) the vast majority of those responses that are going to be returned to the sender by subjects are received within 27 hours of sending. We note this because there is a clear pattern that subjects fall into. (1) They respond immediately; (2) they respond at the beginning of business the next day; or (3) they do not respond. Indeed, registrars in our sample fall well within this pattern of behavior -- the median time to response, for those who respond to us as 8 hours. \n",
    "\n",
    "In the remainder of this section, we demonstrate how slow the response rate was in the time immediately preceeding the widespread notification. \n",
    "\n",
    "White, Nathan and Faller (2015) elect to remove from consideration states that the authors suspected had violated the non-interference or exclusion restriction assumptions in their audit study. We contend that such a determination is unnecessairily consaervative. While it is true that responses that are received after the widespread notification of the study are indeed unique responses, those responses that are received before this notificaiton adhere to all the assumptions necessary to elucidate causal quantities from this research design. \n",
    "\n",
    "To demonstrate that there is little effect of trimming our data to the point that widespread notification was sent to Registrars, in this section we employ a series of duration models. We have not chosen to report our primary causal effects through these models because the difference in means and OLS estimates we *do* report remain unbiased estimators and are considerably simpler to conceptualize. Nonetheless, in this section we demonstrate that moving the censoring point for *non-response* from election day to the time when notification was sent to registrars has no appreciable effect on causal estimates.\n",
    "\n",
    "Second, the *ex ante* expectation for a widespread notification is that registrars will be less likely to respond to an email originating from our sending domain. But, there is little reason to suspect that registrars should behave in this way. We test for differential responsivness pre-and post mailer, and find no evidence to support a theory of differential responsiveness to treatment conditional on this split. \n",
    "\n",
    "We ran pilots in MN, CA, OR, WA, NV. While the timing that we report in the main text excludes these western statea and Minnesota (which was also not utilized by White, Nathan, and Faller (2015)), the results that we estimate are not unique to this composed set of states. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"rateOfResponse\"></a> \n",
    "## 4.A Rate of Responses\n",
    "[toc](#toc)\n",
    "\n",
    "As a descriptive matter, the rate of email responses from registrars had slowed to little more than a trickle at the time that the NASS sent an email to its members. In the next figure, we plot the total number of emails received back to our servers on the y-axis of the plot, and the (logged) time since sending on the x-axis. The rate of response of registrars might be considered as the first derivative of this plot -- or simply understood as the rate of change at a particular time within the study window. \n",
    "\n",
    "As the figure shows, the vast majority of response received by our servers were received quickly after leaving. The median time to response is just over 8 hours, though this distribution is highly right skewed such that a very small number of registrars take a very long time to response (e.g. $\\mu$ time is 14 hours, while the $90^{th}$ percentile time to response is only 11.2 hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "afterNASS\n",
       "   0    1 \n",
       "3646   76 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "censoring.date = \"2016-11-08 00:00:00\"\n",
    "clerk.email.date = \"2016-11-02 20:00:00\" # NASS email to all orgs\n",
    "send.date = \"2016-10-31 09:00:00\"\n",
    "sd = create.survival.data(cd = censoring.date)\n",
    "\n",
    "## also make data on non-surivival set for estimating interaction\n",
    "nineOclock = as.numeric(as.POSIXct(\"2016-11-02 09:00:00\"))\n",
    "oneOclock  = as.numeric(as.POSIXct(\"2016-11-02 13:00:00\"))\n",
    "\n",
    "d = d[ReplyDate != \"\", numReply := as.numeric(as.POSIXct(ReplyDate))]\n",
    "d = d[ , afterNASS := 0]\n",
    "\n",
    "\n",
    "d = d[ReplyDate != \"\"] %>% \n",
    "  .[as.numeric(as.POSIXct(ReplyDate)) > as.numeric(as.POSIXct(clerk.email.date)), \n",
    "      afterNASS := 1]\n",
    "d[ , table(afterNASS)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What was the median time to response, for those responses that we received? Note that this is *not* the time that local election officials use to respond to our stimulus, but rather is the time between when we *sent* the email and when the election official responded. We may have sent the email at 10:00a local time, but the registrar may not have opened the email until after lunch. These two-plus hours are included in the \"time to response\" metric that we report here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "The median time to response was 8.2 hours."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "The 75th percentile time to response was 11.2 hours."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "The 90th percentile time to response was 30.6 hours."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "The mean time to response was 14.1 hours."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = sd[GotResponse == 1 , quantile(x = time, probs = c(.5, .75, .9), na.rm = TRUE)] / 3600 \n",
    "\n",
    "display_html(sprintf(\"The median time to response was %.1f hours.\", x[1]))\n",
    "display_html(sprintf(\"The 75th percentile time to response was %.1f hours.\", x[2]))\n",
    "display_html(sprintf(\"The 90th percentile time to response was %.1f hours.\", x[3]))\n",
    "\n",
    "x = sd[GotResponse == 1 , mean(x = time, na.rm = TRUE)] / 3600 \n",
    "\n",
    "display_html(sprintf(\"The mean time to response was %.1f hours.\", x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pdf(\"./figures-tables/rateOfResponse.pdf\")\n",
    "source('./src/plot_function.R')\n",
    "par(oma=c(0,0,0,0), mar = c(5,7,4,2)+.1, lwd=3)\n",
    "sd[GotResponse == 1] %>% \n",
    "#sd[GotResponse == 1 & !(state %in% c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \"Washington\"))] %>% \n",
    "  .[order(time), .(time = time, count = 1:.N)] %>% \n",
    "  .[ , plot(x=time, y=count, type = \"l\", log = \"x\", \n",
    "           main = \"All Responses\", \n",
    "           xaxt = 'n', xlab = \"Days Since Sending\",\n",
    "           yaxt = 'n', ylab = \"\", ylim = c(0,4000))\n",
    "   ]\n",
    "title(ylab=\"Responses\", line=5)\n",
    "\n",
    "# axis(side = 1, at = c(25200, 1:7*(100800)), labels = paste0(0:7))\n",
    "axis(side = 1, at = c(25200, 1:8*(86400)+25200), labels = paste0(0:8))\n",
    "axis(side = 2, at = c(0, 1000, 2000, 3000, 4000), labels = c(\"0\", \"1,000\", \"2,000\", \"3,000\", \"4,000\"))\n",
    "abline(v = 25200 + (8*86400),\n",
    "       lty = 2, col = \"gray50\")\n",
    "abline(v = 25200 + (2*86400) + 3600 * 20,\n",
    "       lty = 2, col = \"gray50\")\n",
    "text(x=6.3*100800, y=400, labels=\"Election Day\", srt=90)\n",
    "text(x=2.4*100800, y=400, labels=\"NASS Email\", srt=90)\n",
    "# text(x=.35 * 100800, y = 3750, labels=\"(a)\", cex = 2, col = \"grey50\", font=2)\n",
    "\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../figures-tables/rateOfResponse.pdf\" alt=\"Experiment Overview\" width=500>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pdf(\"./figures-tables/rateOfResponsePerCondition.pdf\")\n",
    "source('./src/plot_function.R')\n",
    "par(oma=c(0,0,0,0), mar = c(5,7,4,2)+.1, lwd=3)\n",
    "sd[GotResponse == 1 & ethnic_cue == 0 & !(state %in% c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \"Washington\", \"Michigan\", \"New Hampshire\", \"Colorado\"))] %>% \n",
    "  .[order(time), .(time = time, count = 1:.N)] %>% \n",
    "  .[ , plot(x=time, y=count, type = \"l\", lty = 1, log = \"x\", col = \"grey\", lwd = 5,\n",
    "           main = \"No Interference States:\\n Responses by Condition\", \n",
    "           xaxt = 'n', xlab = \"Days Since Sending\",\n",
    "           yaxt = 'n', ylab = \"\", ylim = c(0,1000))]\n",
    "\n",
    "sd[GotResponse == 1 & ethnic_cue == 1 & !(state %in% c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \"Washington\", \"Michigan\", \"New Hampshire\", \"Colorado\"))] %>% \n",
    "  .[order(time), .(time = time, count = 1:.N)] %>% \n",
    "  .[ , lines(x=time, y=count, lty = 4)]\n",
    "sd[GotResponse == 1 & ethnic_cue == 2 & !(state %in% c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \"Washington\", \"Michigan\", \"New Hampshire\", \"Colorado\"))] %>% \n",
    "  .[order(time), .(time = time, count = 1:.N)] %>% \n",
    "  .[ , lines(x=time, y=count, lty = 3)]\n",
    "sd[GotResponse == 1 & ethnic_cue == 3 & !(state %in% c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \"Washington\", \"Michigan\", \"New Hampshire\", \"Colorado\"))] %>% \n",
    "  .[order(time), .(time = time, count = 1:.N)] %>% \n",
    "  .[ , lines(x=time, y=count, lty = 2)]\n",
    "       \n",
    "title(ylab=\"Responses\", line=4)\n",
    "axis(side = 1, at = c(25200, 1:8*(86400)+25200), labels = paste0(0:8))\n",
    "axis(side = 2, at = c(0, 250, 500, 750, 1000), labels = c(\"0\", \"250\", \"500\", \"750\", \"1,000\"))\n",
    "abline(v = 25200 + (8*86400),\n",
    "       lty = 2, col = \"gray50\")\n",
    "abline(v = 25200 + (2*86400) + 3600 * 20,\n",
    "       lty = 2, col = \"gray50\")\n",
    "text(x=6.3*100800, y=100, labels=\"Election Day\", srt=90)\n",
    "text(x=2.4*100800, y=100, labels=\"NASS Email\", srt=90)\n",
    "# text(x=.35 * 100800, y = 950, labels=\"(c)\", cex = 2, col = \"grey50\", font=2)\n",
    "legend(x = 43200, y = 350, \n",
    "       legend = c(\"White\", \"Black\", \"Latino\", \"Arab\"), cex=1.5,\n",
    "       lty = c(1, 3, 4, 2), lwd = c(5,3,3,3), \n",
    "       col = c(\"grey\", \"black\", \"black\", \"black\"))\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../figures-tables/rateOfResponsePerCondition.pdf\" alt=\"Experiment Overview\" width=500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Demonstrate that using all the states (that is, including MN and the western states does not change the estimates that we report. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pdf(\"./figures-tables/rateOfResponsePerConditionAllStates.pdf\")\n",
    "source('./src/plot_function.R')\n",
    "par(oma=c(0,0,0,0), mar = c(5,7,4,2)+.1, lwd=3)\n",
    "sd[GotResponse == 1 & ethnic_cue == 0] %>% \n",
    "  .[order(time), .(time = time, count = 1:.N)] %>% \n",
    "  .[ , plot(x=time, y=count, type = \"l\", log = \"x\", \n",
    "           main = \"All States: Responses by Condition\", \n",
    "           xaxt = 'n', xlab = \"Days Since Sending\",\n",
    "           yaxt = 'n', ylab = \"\", ylim = c(0,1000), \n",
    "           lty = 1, col = \"grey\", lwd = 5)\n",
    "   ]\n",
    "sd[GotResponse == 1 & ethnic_cue == 1] %>% \n",
    "  .[order(time), .(time = time, count = 1:.N)] %>% \n",
    "  .[ , lines(x=time, y=count, lty = 4)]\n",
    "sd[GotResponse == 1 & ethnic_cue == 2] %>% \n",
    "  .[order(time), .(time = time, count = 1:.N)] %>% \n",
    "  .[ , lines(x=time, y=count, lty = 3)]\n",
    "sd[GotResponse == 1 & ethnic_cue == 3] %>% \n",
    "  .[order(time), .(time = time, count = 1:.N)] %>% \n",
    "  .[ , lines(x=time, y=count, lty = 2)]    \n",
    "\n",
    "\n",
    "    \n",
    "    \n",
    "title(ylab=\"Responses\", line=4)\n",
    "axis(side = 1, at = c(25000, 1:7*(100800)), labels = paste0(0:7))\n",
    "axis(side = 2, at = c(0, 250, 500, 750, 1000), labels = c(\"0\", \"250\", \"500\", \"750\", \"1,000\"))\n",
    "abline(v = 25200 + (2*86400) + 3600 * 20,\n",
    "       lty = 2, col = \"gray50\")\n",
    "abline(v = as.numeric(\n",
    "           as.POSIXct(censoring.date) - sd[ , mean(Date, na.rm = TRUE)]),\n",
    "       lty = 2, col = \"gray50\")\n",
    "text(x=6*100800, y=100, labels=\"Election Day\", srt=90)\n",
    "text(x=2.4*100800, y=100, labels=\"NASS Email\", srt=90)\n",
    "# text(x=.35 * 100800, y = 950, labels=\"(b)\", cex = 2, col = \"grey50\", font=2)\n",
    "legend(x = 43200, y = 350, \n",
    "       lty = c(1,3,4,2), lwd = c(5,3,3,3), \n",
    "       legend = c(\"White\", \"Black\", \"Latino\", \"Arab\"), \n",
    "       col = c(\"grey\", \"black\", \"black\", \"black\"), , cex=1.5)\n",
    "\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../figures-tables/rateOfResponsePerConditionAllStates.pdf\" alt=\"Experiment Overview\" width=500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, show that the estimated effects are the same whether they are estimated in states that did not have spillover, or all states. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "interfere_states <- c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \n",
    "                      \"Washington\", \"Michigan\", \"New Hampshire\", \"Colorado\")\n",
    "\n",
    "d = load.data()\n",
    "\n",
    "interfere_1a <- d[ , felm(GotResponse ~ I(ethnic_cue != 0) + dens_log + \n",
    "                           pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                           obama_margin | factor(blockID)) ]\n",
    "interfere_1b <- d[!(state %in% interfere_states), \n",
    "                  felm(GotResponse ~ I(ethnic_cue != 0) + dens_log + \n",
    "                           pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                           obama_margin | factor(blockID)) ]\n",
    "interfere_2a <- d[ , \n",
    "                 felm(GotResponse ~ factor(ethnic_cue) + dens_log + \n",
    "                      pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                      obama_margin| factor(blockID)) ]\n",
    "interfere_2b <- d[!(state %in% interfere_states) , \n",
    "                 felm(GotResponse ~ factor(ethnic_cue) + dens_log + \n",
    "                      pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                      obama_margin| factor(blockID)) ]\n",
    "interfere_3a <- tracker[ , felm(I(HitTracker == 'True') ~ I(ethnic_cue != 0) + dens_log + \n",
    "                           pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                           obama_margin | factor(blockID)) ]\n",
    "interfere_3b <- tracker[!(state %in% interfere_states), \n",
    "                  felm(I(HitTracker == 'True') ~ I(ethnic_cue != 0) + dens_log + \n",
    "                           pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                           obama_margin | factor(blockID)) ]\n",
    "interfere_4a <- tracker[ , \n",
    "                 felm(I(HitTracker == 'True') ~ factor(ethnic_cue) + dens_log + \n",
    "                      pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                      obama_margin| factor(blockID)) ]\n",
    "interfere_4b <- tracker[!(state %in% interfere_states) , \n",
    "                 felm(I(HitTracker == 'True') ~ factor(ethnic_cue) + dens_log + \n",
    "                      pct_inc2pov_150 + pct_race_black + pct_latino + \n",
    "                      obama_margin| factor(blockID)) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "No Difference in Estimates in Interference States\n",
      "==========================================================================================\n",
      "                                             Dependent variable:                          \n",
      "                   -----------------------------------------------------------------------\n",
      "                                 GotResponse                         HitTracker           \n",
      "                      (1)       (2)       (3)       (4)      (5)     (6)     (7)     (8)  \n",
      "------------------------------------------------------------------------------------------\n",
      "Minority           -0.045*** -0.048***                      0.000   0.000                 \n",
      "                    (0.013)   (0.016)                      (0.000) (0.000)                \n",
      "                                                                                          \n",
      "Latino                                  -0.028*   -0.036*                   0.000   0.000 \n",
      "                                        (0.016)   (0.019)                  (0.000) (0.000)\n",
      "                                                                                          \n",
      "Black                                    0.004    -0.002                    0.000   0.000 \n",
      "                                        (0.016)   (0.019)                  (0.000) (0.000)\n",
      "                                                                                          \n",
      "Arab                                   -0.110*** -0.107***                  0.000   0.000 \n",
      "                                        (0.016)   (0.019)                  (0.000) (0.000)\n",
      "                                                                                          \n",
      "ln(pop dens)       0.152***  0.117***  0.149***  0.112***   0.000   0.000   0.000   0.000 \n",
      "                    (0.029)   (0.035)   (0.029)   (0.034)  (0.000) (0.000) (0.000) (0.000)\n",
      "                                                                                          \n",
      "Pct < 150% Poverty  -0.142    -0.509    -0.109    -0.501    0.000   0.000   0.000   0.000 \n",
      "                    (0.693)   (0.766)   (0.689)   (0.763)  (0.000) (0.000) (0.000) (0.000)\n",
      "                                                                                          \n",
      "Pct Black          -0.498**   -0.227   -0.483**   -0.213    0.000   0.000   0.000   0.000 \n",
      "                    (0.236)   (0.301)   (0.235)   (0.300)  (0.000) (0.000) (0.000) (0.000)\n",
      "                                                                                          \n",
      "Pct Latino          -0.355*   -0.161    -0.367*   -0.179    0.000   0.000   0.000   0.000 \n",
      "                    (0.202)   (0.241)   (0.201)   (0.240)  (0.000) (0.000) (0.000) (0.000)\n",
      "                                                                                          \n",
      "Obama Margin        -0.006     0.058    -0.015     0.053    0.000   0.000   0.000   0.000 \n",
      "                    (0.084)   (0.095)   (0.083)   (0.095)  (0.000) (0.000) (0.000) (0.000)\n",
      "                                                                                          \n",
      "------------------------------------------------------------------------------------------\n",
      "Observations         6,439     4,552     6,439     4,552    3,722   2,783   3,722   2,783 \n",
      "R2                   0.334     0.327     0.341     0.332                                  \n",
      "Adjusted R2          0.109     0.097     0.118     0.104                                  \n",
      "==========================================================================================\n",
      "Note:                                                          *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "stargazer(interfere_1a, interfere_1b, interfere_2a, interfere_2b, \n",
    "          interfere_3a, interfere_3b, interfere_4a, interfere_4b, \n",
    "          type = 'text', \n",
    "          title = 'No Difference in Estimates in Interference States', \n",
    "          label = 'x:tab:spillover',\n",
    "          table.placement = 'p', \n",
    "          header = FALSE,\n",
    "          dep.var.labels = c('GotResponse', 'HitTracker'),\n",
    "          font.size = 'footnotesize',\n",
    "          column.sep.width = '1pt',\n",
    "          covariate.labels = c('Minority', 'Latino', 'Black', 'Arab', \n",
    "                               'ln(pop dens)', 'Pct < 150% Poverty', \n",
    "                               'Pct Black', 'Pct Latino', 'Obama Margin'),\n",
    "          omit.stat = 'ser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = capture.output(\n",
    "    stargazer(interfere_1a, interfere_1b, interfere_2a, interfere_2b, \n",
    "          interfere_3a, interfere_3b, interfere_4a, interfere_4b, \n",
    "          type = 'latex', \n",
    "          out = './figures-tables/interference_states.tex', \n",
    "          title = 'No Difference in Estimates in Interference States', \n",
    "          label = 'x:tab:spillover',\n",
    "          table.placement = 'p', \n",
    "          header = FALSE,\n",
    "          dep.var.labels = c('GotResponse', 'HitTracker'),\n",
    "          font.size = 'footnotesize',\n",
    "          column.sep.width = '1pt',\n",
    "          covariate.labels = c('Minority', 'Latino', 'Black', 'Arab', \n",
    "                               'ln(pop dens)', 'Pct < 150% Poverty', \n",
    "                               'Pct Black', 'Pct Latino', 'Obama Margin'),\n",
    "          omit.stat = 'ser')\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"timeToRespond\"></a>\n",
    "## 4.B Time for Registrars to Respond \n",
    "[toc](#toc)\n",
    "\n",
    "An interesting question that has not been answered in audit studies conducted with government officicals is the amount of time that the audit study requires of each official. This question has its roots both in understanding what is being asked of subjects, and also in understanding the costs of executing the audit study. \n",
    "\n",
    "One of the difficulties with this measurement -- a difficulty that has likely caused previous scholars to overlook this interesting result -- is that it is not possible using standard consumer email to know *when* a subject began the task of addressing the audit. One might consider the response task beginning at the time the audit was initiated and then simply compute the time between receiving a response to the audit task. Clearly, however, this will tend to over-estimate the amount of time required to a task, beacuse it is not the case that *all* units begin working on the task immediately. \n",
    "\n",
    "In our project, we utilized a standard practice in direct-consumer contact and included an image in our email. This image, which was a `1x1` pixel white image at the bottom of our email -- an image commonly called a \"tracking pixel\" loaded from our sending server at the time that the subject opened the email. This provides us with an approximate time that our units began the task, permitting us to calculate a difference between the beginning and completion of the task. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker = fread('./data/replication_tracker.csv')\n",
    "\n",
    "tracker = merge(tracker, d[ , .(registrar_id, RecvDate, ReplyDate, ethnic_cue, GotResponse, blockID)], \n",
    "               by.x = \"registrar_id\", by.y = \"registrar_id\")\n",
    "\n",
    "tracker = tracker[ , timestamp.d := ymd_hms(timestamp)]\n",
    "tracker = tracker[ , RecvDate.d := ymd_hms(RecvDate)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker = tracker[ , seconds.d := RecvDate.d - timestamp.d]\n",
    "tracker = tracker[ , seconds   := as.numeric(seconds.d)]\n",
    "tracker = tracker[ , minutes   := seconds / 60] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we take only the first tracker hit for each LEO. This computes a conservative (long) estimate of the time to complete. We note at this point that we drop *negative* times to complete. These negative times to complete occur because of image caching, mostly on users who are utilizing consumer email clients. We note that LEOs who are using this form of technology are non-random. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker = tracker[ , numHits := .N, by = registrar_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Median Mininum Seconds</th><th scope=col>Median Mininum Minutes</th><th scope=col>Median Maximum Seconds</th><th scope=col>Median Maximum Minutes</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>175 </td><td>2.92</td><td>235 </td><td>3.92</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " Median Mininum Seconds & Median Mininum Minutes & Median Maximum Seconds & Median Maximum Minutes\\\\\n",
       "\\hline\n",
       "\t 175  & 2.92 & 235  & 3.92\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Median Mininum Seconds | Median Mininum Minutes | Median Maximum Seconds | Median Maximum Minutes | \n",
       "|---|\n",
       "| 175  | 2.92 | 235  | 3.92 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Median Mininum Seconds Median Mininum Minutes Median Maximum Seconds\n",
       "1 175                    2.92                   235                   \n",
       "  Median Maximum Minutes\n",
       "1 3.92                  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tracker %>% \n",
    "  .[ , .(min_seconds = min(seconds, na.rm = T), \n",
    "         max_seconds = max(seconds, na.rm = T), \n",
    "         ethnic_cue  = mean(ethnic_cue, na.rm = T)), \n",
    "    keyby = .(registrar_id)] %>% \n",
    "  .[min_seconds > 0 & max_seconds > 0] %>% \n",
    "  .[ , .(\"Median Mininum Seconds\" = median(min_seconds), \n",
    "         \"Median Mininum Minutes\" = as.numeric(median(min_seconds) / 60), \n",
    "         \"Median Maximum Seconds\" = median(max_seconds), \n",
    "         \"Median Maximum Minutes\" = as.numeric(median(max_seconds) / 60))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>pdf:</strong> 2"
      ],
      "text/latex": [
       "\\textbf{pdf:} 2"
      ],
      "text/markdown": [
       "**pdf:** 2"
      ],
      "text/plain": [
       "pdf \n",
       "  2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pdf(file = './figures-tables/timeToResponse.pdf')\n",
    "source('./src/plot_function.R')\n",
    "tracker %>% \n",
    "   .[ , .(min_seconds = min(seconds, na.rm = T), \n",
    "          max_seconds = max(seconds, na.rm = T), \n",
    "          ethnic_cue  = mean(ethnic_cue, na.rm = T)), \n",
    "    keyby = .(registrar_id)] %>% \n",
    "  .[min_seconds > 0 & max_seconds > 0] %>% \n",
    "  .[order(min_seconds)] %>% \n",
    "  .[ , .(t = as.numeric(min_seconds) / 60, count = 1:.N)] %>% \n",
    "  .[ , plot(t, count, type = \"l\", \n",
    "            xlab = \"Minutes Between Open and Response\", xlim = c(0, 120), \n",
    "            ylab = \"Number of Hits\", ylim = c(0, 2155)\n",
    "           )]\n",
    "dev.off()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = \"../figures-tables/timeToResponse.pdf\" width = 500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"noInterruptionDifference\"></a>\n",
    "\n",
    "## 4.C No Difference in Estimated Effect Due to Email \n",
    "[toc](#toc)\n",
    "\n",
    "In duration models, the researcher specifies the point at which a \"failure to convert\" is recorded as such a failure. This class of models, developed in the context of health studies provide unbiased estimates in the face of right censoring. \n",
    "\n",
    "In the design that we registered with EGAP, we specified the censoring date as election day. On election day, any registrar who had not responded to our mailer would be classified as a non-response. This is the date that we use for the main analysis, but we also demonstrate that changing the censoring date does not change the inference drawn from this experiment. In particular, we demonstrate that changing the censoring date to the time of the NASS email to registrars' state organizations neither changes the point estimate of the causal effect, nor does it change the inference of the causal effect of senders' names. \n",
    "\n",
    "To perform this check, we create two survival data sets -- one with the censoring date on election day, and the other with the censoring date at the time of the NASS emailer. On both sets of data, we estiamte Cox Proportional Hazards models with only a term for the `ethnic_cue` sent to registrars. \n",
    "\n",
    "As is evident in the plot, and borne out in the estimated models, the estimates of the causal effect are stable both in terms of location and uncertainty in the models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=============================================================================\n",
      "                                            Dependent variable:              \n",
      "                              -----------------------------------------------\n",
      "                                                   time                      \n",
      "                                  (1)         (2)         (3)         (4)    \n",
      "-----------------------------------------------------------------------------\n",
      "I(ethnic_cue != 0)             -0.134***   -0.138***   -0.127***   -0.127*** \n",
      "                                (0.043)     (0.043)     (0.037)     (0.037)  \n",
      "                                                                             \n",
      "-----------------------------------------------------------------------------\n",
      "Censoring Date                 Election   NASS Email   Election   NASS Email \n",
      "Observations                     4,548       4,548       6,435       6,435   \n",
      "R2                               0.002       0.002       0.002       0.002   \n",
      "Max. Possible R2                 1.000       1.000       1.000       1.000   \n",
      "Log Likelihood                -22,310.000 -22,099.000 -31,241.000 -30,928.000\n",
      "Wald Test (df = 1)             9.720***    10.300***   11.800***   11.700*** \n",
      "LR Test (df = 1)               9.510***    10.100***   11.600***   11.400*** \n",
      "Score (Logrank) Test (df = 1)  9.730***    10.300***   11.800***   11.700*** \n",
      "=============================================================================\n",
      "Note:                                             *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "sd1 = create.survival.data(cd = censoring.date)\n",
    "sd2 = create.survival.data(cd = clerk.email.date)\n",
    "\n",
    "mc1 = sd1[!(state %in% c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \"Washington\", \"Michigan\", \"New Hampshire\", \"Colorado\")), \n",
    "       coxph(Surv(time, survRespond) ~ 1 + I(ethnic_cue != 0))]\n",
    "mc2 = sd2[!(state %in% c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \"Washington\", \"Michigan\", \"New Hampshire\", \"Colorado\")), \n",
    "       coxph(Surv(time, survRespond) ~ 1 + I(ethnic_cue != 0))]\n",
    "mc3 = sd1[ , coxph(Surv(time, survRespond) ~ 1 + I(ethnic_cue != 0))]\n",
    "mc4 = sd2[ , coxph(Surv(time, survRespond) ~ 1 + I(ethnic_cue != 0))]\n",
    "\n",
    "stargazer(mc1, mc2, mc3, mc4, type = \"text\", \n",
    "          add.lines = list(c(\"Censoring Date\", \"Election\", \n",
    "                             \"NASS Email\", \"Election\", \"NASS Email\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=============================================================================\n",
      "                                            Dependent variable:              \n",
      "                              -----------------------------------------------\n",
      "                                                   time                      \n",
      "                                  (1)         (2)         (3)         (4)    \n",
      "-----------------------------------------------------------------------------\n",
      "factor(ethnic_cue)1             -0.097*     -0.102*     -0.076*     -0.072   \n",
      "                                (0.053)     (0.053)     (0.046)     (0.046)  \n",
      "                                                                             \n",
      "factor(ethnic_cue)2             -0.018      -0.025      -0.009      -0.016   \n",
      "                                (0.052)     (0.052)     (0.045)     (0.045)  \n",
      "                                                                             \n",
      "factor(ethnic_cue)3            -0.294***   -0.293***   -0.307***   -0.303*** \n",
      "                                (0.055)     (0.055)     (0.048)     (0.048)  \n",
      "                                                                             \n",
      "-----------------------------------------------------------------------------\n",
      "Censoring Date                 Election   NASS Email   Election   NASS Email \n",
      "Observations                     4,548       4,548       6,435       6,435   \n",
      "R2                               0.008       0.008       0.008       0.008   \n",
      "Max. Possible R2                 1.000       1.000       1.000       1.000   \n",
      "Log Likelihood                -22,297.000 -22,087.000 -31,219.000 -30,908.000\n",
      "Wald Test (df = 3)             35.200***   34.300***   52.400***   49.900*** \n",
      "LR Test (df = 3)               36.500***   35.500***   54.800***   52.200*** \n",
      "Score (Logrank) Test (df = 3)  35.400***   34.500***   52.700***   50.300*** \n",
      "=============================================================================\n",
      "Note:                                             *p<0.1; **p<0.05; ***p<0.01\n"
     ]
    }
   ],
   "source": [
    "fc1 = sd1[!(state %in% c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \"Washington\", \"Michigan\", \"New Hampshire\", \"Colorado\")), \n",
    "       coxph(Surv(time, survRespond) ~ 1 + factor(ethnic_cue))]\n",
    "fc2 = sd2[!(state %in% c(\"Oregon\", \"Minnesota\", \"California\", \"Oregon\", \"Washington\", \"Michigan\", \"New Hampshire\", \"Colorado\")), \n",
    "       coxph(Surv(time, survRespond) ~ 1 + factor(ethnic_cue))]\n",
    "fc3 = sd1[ , coxph(Surv(time, survRespond) ~ 1 + factor(ethnic_cue))]\n",
    "fc4 = sd2[ , coxph(Surv(time, survRespond) ~ 1 + factor(ethnic_cue))]\n",
    "\n",
    "stargazer(fc1, fc2, fc3, fc4, type = \"text\", \n",
    "          add.lines = list(c(\"Censoring Date\", \"Election\", \n",
    "                             \"NASS Email\", \"Election\", \"NASS Email\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = capture.output( \n",
    "    stargazer(mc1, mc2, mc3, mc4, fc1, fc2, fc3, fc4, \n",
    "          type = \"latex\",\n",
    "          out = \"./figures-tables/survivalModels.tex\",\n",
    "          add.lines = list(c(\"Data Subset\", \"Clean\", \"Clean\", \"All\", \"All\", \"Clean\", \"Clean\", \"All\", \"All\"),\n",
    "                           c(\"Censoring Date\", \"Election\", \"Clerk\", \"Election\", \"Clerk\", \"Election\", \"Clerk\", \"Election\", \"Clerk\")), \n",
    "          digits = 2, \n",
    "          omit.stat = c(\"wald\", \"lr\", \"max.rsq\", \"ll\", \"logrank\"), \n",
    "          column.sep.width = \"0pt\", \n",
    "          covariate.labels = c(\"Minority Cue\", \"Latino Cue\", \"Black Cue\", \"Arab Cue\"),\n",
    "          dep.var.caption = NULL, \n",
    "          dep.var.labels.include = FALSE\n",
    "         )\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "To toggle on/off the raw code, click <a href='javascript:code_toggle()'>here</a>."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_html(\"To toggle on/off the raw code, click <a href='javascript:code_toggle()'>here</a>.\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
